Bytes are still tokens. You’d still have tokenization issues, just new and likely worse ones. Note digits are tokenized specially in GPT-4 as one token per pair. Even in that special case they didn’t go down to single digits.
@goodside
-
Satirical take on RAG papers and retrieval prompting
By
–
every RAG paper starts like We achieve SoTA performance on this challenging benchmark via a retrieval prompting pipeline paired with a synthetically generated knowledge base of interpreter-verified solutions, a method we call RAG Against the Machine.
-
Many-shot examples and their effect on in-context learning
By
–
I’m not convinced many-shot examples degrade performance. Artificial distractor text will, as will long inputs, but many (diverse, reasonable) examples in traditional k-shot ICL should strictly help performance at the (now much lower) expense of latency/cost.
-
Why “fine-tuning” is a misleading term for LLM updates
By
–
“Fine-tuning” is an unfortunate term. FT makes “fine” updates to model weights, but these can effect radical changes in almost any aspect of LLM behavior — to a user it’s more like “retraining.” This makes any use of FT’s lay definition (“tweaking”) in AI contexts confusing.
-
Models using cached tokens greatly reduce cost and latency
By
–
This feature really does change things. Gemini had cache, but only for costs — Claude claims 90% less cost *and* 85% less latency for cached tokens. Even if it can’t replace all SFT, it’s so much easier to iterate on you probably want to exhaust caching-based strategies first.
-
GPT-4’s RL vs Imitation: A Student Playing a Teacher’s Quirky Grading System
By
–
This was very noticeable in GPT-4 around its release and anecdotally it’s gotten better since. My complete armchair guess is this is RL rather than imitation — it feels like a student playing to their teacher’s quirky grading system.
-
GPT-4o blurts wrong answer, then self-corrects; structured output tangential
By
–
It helps, but in this case GPT-4o often blurts out the wrong answer at the start but then realizes and fixes the mistake as it talks. If you grade liberally in that case, most naive prompts asking for reasoning will work fine. Structured Output is slightly tangential here.
-
OpenAI docs include ‘9.11 > 9.9’ solved by JSON output
By
–
"9.11 > 9.9" is now in the OpenAI docs, as a problem solved by requesting JSON structured output to separate final answers from supporting reasoning:
-
Human bias affects model’s count of R’s in strawberry
By
–
I’m just guessing this is the reason but the modal answer from all human statements about the count of R’s in “strawberry” might actually be 2. So it does it because we do.