In NLP we breaking down text into smaller pieces, commonly known as tokens, usually words or subwords. An essential initial step in that transforms raw text into a format that can be consumed by embedding(vectors) generators. Here's how we do it in a typical Text Dataset Class:

→ View original post on X — @akshay_pachaar