Of course, when using an open source model (Vicuna, Dolly) that model name is not in the tiktoken wrapper Previously, this would raise an error We've now updated this to: (1) fallback to a default tokenizer, (2) allow users to specify model name to use for tiktoken

→ View original post on X — @hwchase17,