Seeing as I published my Tokenizer video yesterday, I thought it could be fun to take a deepdive into the Gemma tokenizer. First, the Gemma technical report [pdf]: https://
storage.googleapis.com/deepmind-media
/gemma/gemma-report.pdf
… says: "We use a subset of the SentencePiece tokenizer (Kudo and Richardson, 2018) of
@karpathy
-
Deep dive into Gemma tokenizer and SentencePiece implementation
By
–
-
Data Poisoning Paper: Practical Applications and Implications
By
–
also quite enjoyed his recent paper on data poisoning is practical.
-
MinBPE: Minimal Byte Pair Encoding Implementation for LLM Tokenization
By
–
Also, releasing new repository on GitHub: minbpe
Minimal, clean, code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. https://
github.com/karpathy/minbpe In the video we essentially build minbpe from scratch.
Don't miss the http://
exercise.md to build your -
New Lecture: Building GPT Tokenizer with Byte Pair Encoding
By
–
New (2h13m ) lecture: "Let's build the GPT Tokenizer" Tokenizers are a completely separate stage of the LLM pipeline: they have their own training set, training algorithm (Byte Pair Encoding), and after training implement two functions: encode() from strings to tokens, and
-
Tokenization: The Root of LLM Problems and Quirks
By
–
We will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
-
Andrej Karpathy Announces Departure from OpenAI
By
–
Hi everyone yes, I left OpenAI yesterday. First of all nothing "happened" and it’s not a result of any particular event, issue or drama (but please keep the conspiracy theories coming as they are highly entertaining :)). Actually, being at OpenAI over the last ~year has been
-
Effort to implement and test a feature or project
By
–
definitely, would love to try to get this working:https://t.co/7RASu4dzUG
— Andrej Karpathy (@karpathy) 4 février 2024definitely, would love to try to get this working:
-
Positive feedback on Obsidian with image pasting improvement suggestion
By
–
Ty, @obsdmd looks very very nice based on some quick testing just now! Works perfect, support for code, math. My only piece of feedback is around image pasting. I do a lot of taking screenshots of something, and I want to paste them in fast. Currently, it creates an ugly filename
-
Reconsidering AGI and Self-Driving: Focusing on Technical Work
By
–
It was an okay post. I think I wasn't careful enough with some nuances. Anyway, reading some of the response I realized that both topics (AGI and self-driving) are so charged and toxic right now and I should just stick to technical things
-
Appreciation for llama2.c project and its helpful contribution
By
–
so cool 😀
(& very happy to see llama2.c referenced as helpful )