Ty for rerunning! Curious btw they cite 84.9 in the release, why is it 82.9 here under “Original”? Maybe you know more about the subtleties here
@karpathy
-
Request for Updated Comparison of Latest AI Models Across Organizations
By
–
Would be interesting to see this updated with the latest models from all orgs
-
Claude 3 Tokenization Challenge: Stylistic but Contains Subtle Hallucinations
By
–
Claude 3 takes on the Tokenization book chapter challenge 🙂 context: https://t.co/yRaeTbkblY
— Andrej Karpathy (@karpathy) 4 mars 2024
Definitely looks quite nice, stylistically!
If you look closer there are a number of subtle issues / hallucinations. One example there is a claim that "hello world" tokenizes into 3… https://t.co/mMjK27Uk2yClaude 3 takes on the Tokenization book chapter challenge 🙂 context: https://
x.com/karpathy/statu
s/1760740503614836917
… Definitely looks quite nice, stylistically! If you look closer there are a number of subtle issues / hallucinations. One example there is a claim that "hello world" tokenizes into 3 -
Understanding Consistent Behavior Across Independent Language Models
By
–
I’d love to understand this better too… I thought it was just a quirk of the specifics of labeling instructions, but then multiple (what I think should be mostly independent) language models seem to all do this.
-
Translating lecture content to book format remains challenging
By
–
It's very cool that something sensible comes out!
Here is the "ground truth" I wrote for ~hour yesterday translating just the first few minutes into book-like format: https://
github.com/karpathy/minbp
e/blob/master/lecture.md
…
Which I think is still quite hard. I may still be useful for a few months 🙂 -
Manual transcription of video to markdown taking considerable time
By
–
Ok I wrote the following example of what I am imagining: https://
github.com/karpathy/minbp
e/blob/master/lecture.md
… This is me doing this task manually, of watching the video and translating it to a markdown post. I only made it to the ~4min mark in the video (i.e. 3% done) and this already took about 30 minutes -
Converting Video Lectures into Written Content Using LLMs
By
–
Fun LLM challenge that I'm thinking about: take my 2h13m tokenizer video and translate the video into the format of a book chapter (or a blog post) on tokenization. Something like: 1. Whisper the video
2. Chop up into segments of aligned images and text
3. Prompt engineer an LLM -
Gemma’s 256K Token Context: Analyzing Language Distribution Patterns
By
–
Keep in mind Gemma has a 256K tokens, so all text could be quite a bit shorter because there are so many merges. The interesting analysis here is to look at the *distribution* of token counts across different languages, and compare it to that same distribution for previous.
-
Technical Accessibility: The Impact of Video Demonstrations
By
–
# on technical accessibility One interesting observation I think back to often:
– when I first published the micrograd repo, it got some traction on GitHub but then somewhat stagnated and it didn't seem that people cared much.
– then I made the video building it from scratch,