this is neat. Anthropic never released the tokenizer for Claude 3 (I'm not sure why?). to tokenize a string, you can just ask Claude to repeat it, and store the tokens as they're generated. probably the most computationally expensive tokenizer ever created, but it works

→ View original post on X — @jxmnop