Ok I wouldn't say it was "wrong", just "misleading" :). The idea of having a vector stored at each node in the graph, and the vectors communicating via weighted sums over directed edges I think is all well and sound all by itself, at this level of description. It's misleading
@karpathy
-
Brain Computing Power Compared to NVIDIA A100 GPU
By
–
The naive napkin math would go something like 1 brain ~= 1e11 neurons * 1e4 synapses * 1e1 fires/s = 1e16 FLOPS (i.e. 10 petaflops) NVIDIA A100 = 312e12 peak FLOPS, in-practice achievable utilization may be let’s say 50%, i.e. 156e12. Dividing you get 1 brain ~= 1e16 / 156e12 =
-
LLMs’ Killer App is Scarlett Johansson, Not Math
By
–
The killer app of LLMs is Scarlett Johansson. You all thought it was math or something
-
OpenAI releases multimodal model with text audio vision capabilities
By
–
They are releasing a combined text-audio-vision model that processes all three modalities in one single neural network, which can then do real-time voice translation as a special case afterthought, if you ask it to.
— Andrej Karpathy (@karpathy) 13 mai 2024
(fixed it for you) https://t.co/0y36OId88hThey are releasing a combined text-audio-vision model that processes all three modalities in one single neural network, which can then do real-time voice translation as a special case afterthought, if you ask it to. (fixed it for you)
-
Leading AI Safety in Major Paradigm-Shifting Deployments Today
By
–
Just one of the very few people both in charge of and in thick of the practical AI safety of today in the biggest, paradigm shifting deployments of AI today…
-
Learning CUDA: Resources Gap Between Theory and Practice
By
–
I read this book and then I was surprised that I still understood so little of the kernels that started to appear as llm.c contributions, beating mine. It's a pretty good 101 intro. Learning CUDA is like that horse meme, all the learning resources you can find on the left, then
-
Prompting vs Fine-tuning for Better Model Performance
By
–
You can kind of do this already by a bit of prompting, but probably you're right that if you target this as a finetune it might come out better.
-
Estimating GPT Grades for Written Content Quality
By
–
Anyone else find themselves estimating the "GPT grade" of things you hear/read? When something is poorly written or generic, it's "GPT-2 grade" content. When something is lit, you can complement it as being "GPT-7 grade" etc. This reminds me of a fun side project I had saved for
-
Token Encoding and Decoding: Asymmetric Complexity in LLMs
By
–
decoding (tokens -> string) is just lookup table and string concat. encoding (string -> tokens) is a pain. For sentencepiece I *think* llama2.c has a simple implementation that probably works but I'm not 100% sure: https://
github.com/karpathy/llama
2.c/blob/master/run.c#L452
… For tiktoken-style, the problem is the -
Deep dive into tokenization vulnerabilities across multiple language models
By
–
Nice new read on tokenization!
You've heard about the SolidGoldMagikarp token, which breaks GPT-2 because it was present in the training set of the Tokenizer, but not the LLM later. This paper digs in in a lot more depth and detail, on a lot more models, discovering a less