I always learn a lot more from in-depth analysis of few random cases over dashboards of aggregate statistics across all cases. Both projections can be helpful but the latter is disproportionately pervasive.
@karpathy
-
Reinforcement Learning Field of View Limitations and Policy Optimization
By
–
Yep I think RL is misleading in that it restricts field of view. Eg like you mentioned you can imagine review/reflect doing a lot more – building tools for later use, or actively tuning the distribution for what to try next (instead of just sampling from policy independently as
-
Scaling Reinforcement Learning: Intermediate Gains and Future Limitations
By
–
Scaling up RL is all the rage right now, I had a chat with a friend about it yesterday. I'm fairly certain RL will continue to yield more intermediate gains, but I also don't expect it to be the full story. RL is basically "hey this happened to go well (/poorly), let me slightly
-
Vision VAE as Ultimate Tokenizer: Unicode Efficiency
By
–
Very cool work direction but also fair question.
I wonder if ultimately is a little vision patch VAE the ultimate "tokenizer"? Unicode + UTF-8 is just too high description length. -
Rethinking Research Format for LLM Consumption Over PDFs
By
–
I often rant about how 99% of attention is about to be LLM attention instead of human attention. What does a research paper look like for an LLM instead of a human? It’s definitely not a pdf. There is huge space for an extremely valuable “research app” that figures this out.
-
AI Knowledge: From Human-Readable to Machine-Native Formats
By
–
As AI advances, our contribution is more and more original knowledge – meaning something that can’t be inferred from what exists digitally already by reasoning. Something like the result of an experiment. Maybe it should be written more natively for AIs instead of people, eg PDF
-
Knowledge enhances understanding of world and technology
By
–
Knowledge makes the world so much more beautiful.
-
Code Bloat: Why Software Lacks Biology’s Natural Regularization
By
–
Indeed, huge dependency epidemic out there. In biology, code is energetically expensive so genomes have natural regularization. In software the cost of code is lower so it bloats like crazy into brittle mess.
-
Building thriving open source communities through bacterial code principles
By
–
How to build a thriving open source community by writing code like bacteria do . Bacterial code (genomes) are: – small (each line of code costs energy)
– modular (organized into groups of swappable operons)
– self-contained (easily "copy paste-able" via horizontal gene -
nanoGPT: From Teaching Tool to Self-Improvement Benchmark
By
–
Love this project: nanoGPT -> recursive self-improvement benchmark. Good old nanoGPT keeps on giving and surprising 🙂 – First I wrote it as a small little repo to teach people the basics of training GPTs.
– Then it became a target and baseline for my port to direct C/CUDA