We will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.
@karpathy
-
Andrej Karpathy Announces Departure from OpenAI
By
–
Hi everyone yes, I left OpenAI yesterday. First of all nothing "happened" and it’s not a result of any particular event, issue or drama (but please keep the conspiracy theories coming as they are highly entertaining :)). Actually, being at OpenAI over the last ~year has been
-
Effort to implement and test a feature or project
By
–
definitely, would love to try to get this working:https://t.co/7RASu4dzUG
— Andrej Karpathy (@karpathy) 4 février 2024definitely, would love to try to get this working:
-
Positive feedback on Obsidian with image pasting improvement suggestion
By
–
Ty, @obsdmd looks very very nice based on some quick testing just now! Works perfect, support for code, math. My only piece of feedback is around image pasting. I do a lot of taking screenshots of something, and I want to paste them in fast. Currently, it creates an ugly filename
-
Reconsidering AGI and Self-Driving: Focusing on Technical Work
By
–
It was an okay post. I think I wasn't careful enough with some nuances. Anyway, reading some of the response I realized that both topics (AGI and self-driving) are so charged and toxic right now and I should just stick to technical things
-
Appreciation for llama2.c project and its helpful contribution
By
–
so cool 😀
(& very happy to see llama2.c referenced as helpful ) -
The Real AI Talent: Unknown Innovators Deep in Organizations
By
–
In AI at least, the real 30 under 30 imo you have never heard of. They are 5 layers down the org chart from the CEO. They are usually not on Twitter, they have an unmaintained LinkedIn, they don’t go on podcasts, and they maybe published at one point but don’t do so anymore. They
-
LLM Training Data: From Text to Human Feedback Validation
By
–
Stop, this has nothing to do with neuralink haha.
Anway that's only Stage 1 of enlightenment.
Stage 2 of enlightenment is that the ideal training data for an LLM is not training data at all.
It's the thumbs up you get from someone who reads it.
But you make do with what there is. -
Ideal LLM Training Data: Internal Thoughts and Edits Over Final Text
By
–
The ideal training data for an LLM is not what you wrote. It's the full sequence of your internal thoughts and all the individual edits while you wrote it.
But you make do with what there is. -
Flow Engineering Boosts Code Generation Performance Significantly
By
–
Prompt engineering (or rather "Flow engineering") intensifies for code generation. Great reading and a reminder of how much alpha there is (pass@5 19% to 44%) in moving from a naive prompt:answer paradigm to a "flow" paradigm, where the answer is constructed iteratively.