I often rant about how 99% of attention is about to be LLM attention instead of human attention. What does a research paper look like for an LLM instead of a human? It’s definitely not a pdf. There is huge space for an extremely valuable “research app” that figures this out.
@karpathy
-
AI Knowledge: From Human-Readable to Machine-Native Formats
By
–
As AI advances, our contribution is more and more original knowledge – meaning something that can’t be inferred from what exists digitally already by reasoning. Something like the result of an experiment. Maybe it should be written more natively for AIs instead of people, eg PDF
-
Knowledge enhances understanding of world and technology
By
–
Knowledge makes the world so much more beautiful.
-
Code Bloat: Why Software Lacks Biology’s Natural Regularization
By
–
Indeed, huge dependency epidemic out there. In biology, code is energetically expensive so genomes have natural regularization. In software the cost of code is lower so it bloats like crazy into brittle mess.
-
Building thriving open source communities through bacterial code principles
By
–
How to build a thriving open source community by writing code like bacteria do . Bacterial code (genomes) are: – small (each line of code costs energy)
– modular (organized into groups of swappable operons)
– self-contained (easily "copy paste-able" via horizontal gene -
nanoGPT: From Teaching Tool to Self-Improvement Benchmark
By
–
Love this project: nanoGPT -> recursive self-improvement benchmark. Good old nanoGPT keeps on giving and surprising 🙂 – First I wrote it as a small little repo to teach people the basics of training GPTs.
– Then it became a target and baseline for my port to direct C/CUDA -
The Race for LLM Cognitive Core: Always-On Personal Computing Kernel
By
–
The race for LLM "cognitive core" – a few billion param model that maximally sacrifices encyclopedic knowledge for capability. It lives always-on and by default on every computer as the kernel of LLM personal computing.
Its features are slowly crystalizing: – Natively multimodal -
Preference for Scalable Turn-the-Crank Algorithms with Increased Compute
By
–
I sometimes try to explain it as a statement of preference for "turn the crank" algorithms. When you're eventually given more compute (faster crank), you shouldn't have to touch anything at all, you just crank faster to make better. You can (and probably locally should) knowingly
-
Strong Regularization Prevents RLHF Model Degradation
By
–
May your regularizer be strong, lest you RLHF to slop.
-
Distinguishing between simple prompts and complex application contexts
By
–
Haha I'm not trying to coin a new word or something. I just think people's use of "prompt" tends to (incorrectly) trivialize a rather complex component. You prompt an LLM to tell you why the sky is blue. But apps build contexts (meticulously) for LLMs to solve their custom tasks.