Haha "Llama 270B". We have work to do 🙂
@karpathy
-
Translating lecture content to book format remains challenging
By
–
It's very cool that something sensible comes out!
Here is the "ground truth" I wrote for ~hour yesterday translating just the first few minutes into book-like format: https://
github.com/karpathy/minbp
e/blob/master/lecture.md
…
Which I think is still quite hard. I may still be useful for a few months 🙂 -
Manual transcription of video to markdown taking considerable time
By
–
Ok I wrote the following example of what I am imagining: https://
github.com/karpathy/minbp
e/blob/master/lecture.md
… This is me doing this task manually, of watching the video and translating it to a markdown post. I only made it to the ~4min mark in the video (i.e. 3% done) and this already took about 30 minutes -
Converting Video Lectures into Written Content Using LLMs
By
–
Fun LLM challenge that I'm thinking about: take my 2h13m tokenizer video and translate the video into the format of a book chapter (or a blog post) on tokenization. Something like: 1. Whisper the video
2. Chop up into segments of aligned images and text
3. Prompt engineer an LLM -
Gemma’s 256K Token Context: Analyzing Language Distribution Patterns
By
–
Keep in mind Gemma has a 256K tokens, so all text could be quite a bit shorter because there are so many merges. The interesting analysis here is to look at the *distribution* of token counts across different languages, and compare it to that same distribution for previous.
-
Technical Accessibility: The Impact of Video Demonstrations
By
–
# on technical accessibility One interesting observation I think back to often:
– when I first published the micrograd repo, it got some traction on GitHub but then somewhat stagnated and it didn't seem that people cared much.
– then I made the video building it from scratch, -
Deep dive into Gemma tokenizer and SentencePiece implementation
By
–
Seeing as I published my Tokenizer video yesterday, I thought it could be fun to take a deepdive into the Gemma tokenizer. First, the Gemma technical report [pdf]: https://
storage.googleapis.com/deepmind-media
/gemma/gemma-report.pdf
… says: "We use a subset of the SentencePiece tokenizer (Kudo and Richardson, 2018) of -
Data Poisoning Paper: Practical Applications and Implications
By
–
also quite enjoyed his recent paper on data poisoning is practical.
-
MinBPE: Minimal Byte Pair Encoding Implementation for LLM Tokenization
By
–
Also, releasing new repository on GitHub: minbpe
Minimal, clean, code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. https://
github.com/karpathy/minbpe In the video we essentially build minbpe from scratch.
Don't miss the http://
exercise.md to build your -
Tokenization: The Root of LLM Problems and Quirks
By
–
We will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.