Surprising because this is showing an open weights 70B model at GPT-4 level (for any prompt I may wish to ask)
@karpathy
-
Memory optimization strategies for model training
By
–
didn't realize it was that easy, will take a look at; you can also try decreasing the batch size all the way down to 1, or then also decreasing sequence length until it fits, but you're compromising on max context length that way.
-
The Three Essential Elements of High-Quality Datasets
By
–
The 3 key elements of a good dataset: 1. quality
2. diversity
3. quantity You can only easily measure the last one but the performance is a sensitive function of all three. Super interesting topic ty for #longread :)! -
LLMs as Arrays: Meditation on Entropy and Truth
By
–
As I emerged from meditation it dawned on me that LLMs are just one array of floats and a while loop over some super simple arithmetic on its elements. It is entropy that is the root of suffering. It's by deleting the superfluous that we uncover truth. And thus I was enlightened.
-
Nostalgic complaint about modern token streaming versus sequential watching
By
–
ugh kids these days! back in my days we used to watch the tokens stream one at a time and wait for the output.
-
GPT-2 Reproduction with Increased Channel Size and Memory Optimization
By
–
We want to do a full GPT-2 repro, at channel size 1600 this is 2.1X higher C. And we'll want to ~max out batch dim to fit in memory too. So the "easy times" will be over soon.
-
GPT-2 Activation Memory and GPU Cache Analysis
By
–
Makes sense, in GPT-2 (124M) case we're currently doing B=4, T=1024, C=768 => 3M activations @ float32 => 12MB. A100 L2 cache is 40MB, and even L1, at 192KB/SM with 108 SMs => ~= 20MB (wow, that's more than I expected). The pleasures of smaller networks and caches…
-
Kernel optimization attempts yield minimal performance gains
By
–
added under kernel4 https://
github.com/karpathy/llm.c
/commit/cb791c4ef58d45d58e5af624b0ed41439ac7aeff
…
a bit surprised to only see ~1-2% out of it, which then washes out in training, as the layernorm is not a top-ranking time kernel. Also tried float4 and unrolling but that didn't improve it too much bleh -
Nostalgic Reflection on Past Project with Future Rewrite Plans
By
–
oh my god blast from the past maybe one day i shall do a re-write of this project.
i am imagining efficient, batched training + inference running the brain of all the little bots… -
GPT-2 as the Hello World of Large Language Models
By
–
GPT-2 is the "hello world" of LLMs I think (there must be a better analogy… err MOS 6502? xv6?), so that's why I started there. And it has a proper paper, weights released and available, and a lot is known about it. At this point it is an artifact of historical significance.