Exciting! Big congrats!! (And I can't wait for Llama 4, make it awesome pls!)
@rasbt
-
Character-level tokenization inefficiency and token limit constraints
By
–
You could 100% do that. Actually, I have a paragraph on that in the post: the prob is that if you have a text of 100 characters, that would be 100 tokens (instead of ~20-30 tokens). In other words, it would be wasteful because you won't be able to input longer texts into the LLM.
-
Reader Project Training Tokenizer from Scratch on Nepali
By
–
And here's a great reader project who trained a tokenizer from scratch on Nepali:
-
Skeptical Take on Recent Research Paper Aging
By
–
I am tempted…But we will see if this ages well (just sayin https://
arxiv.org/abs/2412.09871 ) -
Tokenizers deserve their own comprehensive book
By
–
Ah yes, it didn't quite fit into the page limits of chapter 2. And I think tokenizers could be a whole book by itself haha
-
LLMs from Scratch book uses tiktoken over BPE for practical efficiency
By
–
Context: While my LLMs from Scratch book explains and uses BPE tokenizers, I opted to use the highly performant tiktoken library (which is used for GPT-4 and now also used for Llama 3 as well) for practical purposes: the book focuses on LLMs rather than tokenizer development.
-
BPE Tokenizer Implementation from LLMs from Scratch Book
By
–
A reader recently shared a resource on training tokenizers for new languages which reminded me I originally wrote a BPE Tokenizer for my “LLMs from Scratch” book but never shared it! If you are looking a weekend project, here you go:
-
Workflow pragmatisme: clé de la réalisation du livre LLM
By
–
Maybe not the most elegant solution, but honestly, without that workflow, the LLM book would never have been written (at least not this decade) 😀
(Winds of Winter author, pls take note :D) -
Llama 3, Inference Scaling, and Multimodal LLM Paradigms 2025
By
–
The TOC as a tl;dr
1. Of course, the Llama 3 series as one of the main open-weight model releases this year, featuring more sophisticated pre- and post-training pipelines
2. Techniques for improving LLMs by scaling inference-time compute
3. The different multimodal LLM paradigms