I agree and I clarify below I’m only claiming to disprove the (common I think) misunderstanding that LLMs normally do count tokens well, just not letters as a special case
PROMPT ENGINEERING
-
On LLM tokenization and misgeneralization
By
–
Some people do take “LLMs see tokens not letters” at face value — i.e. the mental model is LLMs can of course count repetitions of a *token*, but a naive tokenizer makes letters an unnecessarily difficult task. Misgeneralization during training is different and more plausible.
-
Tokenization isn’t the only cause of LLM counting errors
By
–
Many LLM errors *are* downstream of tokenization oddities, surely. But the folk wisdom that two-R’s-in-strawberry mistakes happen *only* because the letters in “strawberry” are combined into longer prompt tokens before the model can count them is demonstrably not true.
-
LLM counting errors: ChatGPT-4o miscounts repeated tokens
By
–
Another clue: LLM counting errors occur even when counting words instead of letters. Here, ChatGPT 4o miscounts just 4 repetitions of “horse” in a 3-line text, despite care to ensure each “horse” maps consistently to the same single token (20998, “horse” with a leading space):
-
Similar responses from o1-mini and Claude 3.5 Sonnet
By
–
Similar responses responses to the same prompt can be seen from o1-mini and Claude 3.5 Sonnet. Other models left as an engagement-bait exercise for the reader.
-
GPT-4o tokenization splits ‘strawberry’ into single-character tokens
By
–
In both the prompt and response above, each letter in “strawberry” is assigned by the GPT-4o tokenizer to a separate single-character token, instead of being combined into subword-length tokens like “st” “raw” “berry” etc.:
-
Dispelling letter-counting myth by asking ChatGPT about strawberry
By
–
Dispelling a popular myth about letter-counting issues in LLMs by asking ChatGPT: h·o·w m·a·n·y R’s a·r·e i·n “s·t·r·a·w·b·e·r·r·y”?
-
Remixing AI Generations from Gallery: Cool Feature Guide
By
–
one of the coolest features is being able to remix generations from the gallery!
— KREA AI (@krea_ai) 19 septembre 2024
here's how you can do it. pic.twitter.com/sdZPr3xf9yone of the coolest features is being able to remix generations from the gallery! here's how you can do it.
-
ToolComp: New Tool Use Benchmark with Process Supervision
By
–
This is a new benchmark (ToolComp) that encompasses a broader range of Tool Use scenarios than prior benchmarks. Uniquely, we are further evaluating the models utilizing process supervision labels. We also split the benchmark into Enterprise and Chat use cases to differentiate
-
Models Struggle with Instruction Following and Hallucinated Information
By
–
In our error analysis, we found that most models continued to struggle with: – Final Answer Missing Information
– Hallucinated Information This indicates a widespread issue with precise instruction following, especially in prompts requiring detailed information and intermediary