Sometimes I see papers with hyperparameter sweeps over 0.001, 0.003, 0.006, 0.01, etc. Many hyperparameters are better expressed in negative integral log2. Small values like learning rates directly, and values close to 1 like EMA factors and TD lambda / gamma with 1-2**val. It
RESEARCH
-
Model Quantization for Inference: Technical Questions
By
–
Thank you! Notably I didn't yet include model quantization for inference. I have questions 🙂
-
Hugging Face Datasets Critical Infrastructure for AI Model Training
By
–
Ty! huggingface work/infra/datasets are critical to projects like nanochat – to be accurate the source code of nanochat (e.g. at the $100 tier) is ~8KB of Python and ~30GB of fineweb/smoltalk.
-
QK Norm and Embedding Weight Sharing in Model Optimization
By
–
Very early on in the project I did a small run with/without QK norm and found that it helped. Same for the embedding weight sharing. I'll retry! I'm not tied to any details of the model and they weren't chosen any more carefully than a single run, I spent most of the time just
-
Advanced Llama Architecture: Rotary Embeddings and ReLU² MLP
By
–
> llama-like architecture
> dense transformer > rotary only (no positional embeddings) > qk norm > untied embedding/unembedding > norm after token embedding > relu² mlp > no biases in linears > no learnable rmsnorm params > mqa > logit softcap > optimizer = -
Karpathy’s Fundamental AI Resource: Essential Guide for Beginners
By
–
this is an amazing gift to everyone who's just starting straight to the point and has the fundamentals covered in the right order (no surprise though since it's karpathy)
-

Karpathy releases new LLM training project under 8000 lines
By
–
> karpathy just released a new project > in it, and in under 8000 lines of code you get to: > train the tokenizer using a new rust implementation
> pretrain a transformer llm on fineweb, evaluate core score across a number of metrics > midtrain on user-assistant -
LLM101n Experiences Significant Scope Creep Issues
By
–
LLM101n is currently experiencing a lot of scope creep
-
From Zero to Attention Mechanisms: Demystifying LLM Knowledge
By
–
– you are
– a random CS grad with 0 clue how LLMs work
– get tired of people gatekeeping with big words and tiny GPUs
– decide to go full monk mode
– 2 years later i can explain attention mechanisms at parties and ruin them – here’s the forbidden knowledge map
– top to bottom,