If you are using a model from someone else or creating an RAG-based application, be sure to use the same tokenizer (the one most appropriate to your task) for both building your dataset/training and querying (using) the model.

→ View original post on X — @whats_ai,