Ah yes, it didn't quite fit into the page limits of chapter 2. And I think tokenizers could be a whole book by itself haha
@rasbt
-
LLMs from Scratch book uses tiktoken over BPE for practical efficiency
By
–
Context: While my LLMs from Scratch book explains and uses BPE tokenizers, I opted to use the highly performant tiktoken library (which is used for GPT-4 and now also used for Llama 3 as well) for practical purposes: the book focuses on LLMs rather than tokenizer development.
-
BPE Tokenizer Implementation from LLMs from Scratch Book
By
–
A reader recently shared a resource on training tokenizers for new languages which reminded me I originally wrote a BPE Tokenizer for my “LLMs from Scratch” book but never shared it! If you are looking a weekend project, here you go:
-
Workflow pragmatisme: clé de la réalisation du livre LLM
By
–
Maybe not the most elegant solution, but honestly, without that workflow, the LLM book would never have been written (at least not this decade) 😀
(Winds of Winter author, pls take note :D) -
Llama 3, Inference Scaling, and Multimodal LLM Paradigms 2025
By
–
The TOC as a tl;dr
1. Of course, the Llama 3 series as one of the main open-weight model releases this year, featuring more sophisticated pre- and post-training pipelines
2. Techniques for improving LLMs by scaling inference-time compute
3. The different multimodal LLM paradigms -
Noteworthy AI Research Papers 2024 Part 2 Review
By
–
Catching up on improving LLMs by scaling inference-time compute, new LLM scaling laws for precision, and more…
"Noteworthy AI Research Papers 2024 Part 2" is now formatted and published. It was certainly an eventful year for this fast-moving field: -
Classic LLMs Like ChatGPT Explanation Capabilities
By
–
Classic LLMs like ChatGPT will probably happily explain
-
Rejection Sampling and SFT with 17k Examples Discussion
By
–
I mean, rejection sampling and SFT on just 17k examples, what's not to like ?
-
Sky-T1-32B: Open-Source Reasoning Model Rivals o1-Preview Performance
By
–
"Sky-T1-32B-Preview, our reasoning model that performs on par with o1-preview on popular reasoning and coding benchmarks."
That was quick! Is this already the Alpaca moment for reasoning models? Source: https://
novasky-ai.github.io/posts/sky-t1/