A new token based text splitter Splitting text is another big part of applications. Mostly has been done on a character level up until now @kahkeng added a new way to split text, based on tokens Docs: https://
langchain.readthedocs.io/en/latest/modu
les/utils/combine_docs_examples/textsplitter.html#token-text-splitter
…

→ View original post on X — @langchain,