I quite like it as a nice/intuitive testbed of in-context learning, and the experiments around example order, label names, label flipping, etc., which give a sense of the strength of the prior, and ICL as an optimizer. Does the performance here also correlate with other LLM
@karpathy
-
LLMs as Memory-Based Answerers Without Reference Materials
By
–
The way to think about asking a factual question to an LLM is that it's a bit like asking a person who read about the topic previously, but they are not allowed to reference any material and have to answer just from memory. LLMs are a lot better at memorizing than humans, but the
-
LLM Interfaces Need Built-in Proof and Verification Features
By
–
One built-in UI/UX feature of LLM interfaces I'd love is proof. I almost always do this manually – for example if the LLM recommends running some commands with some switches, I manually look up and verify the API in the docs to make sure those switches are correct and that I
-
Micrograd: 94 Lines to Train Neural Networks
By
–
These 94 lines of code are everything that is needed to train a neural network. Everything else is just efficiency. This is my earlier project Micrograd. It implements a scalar-valued auto-grad engine. You start with some numbers at the leafs (usually the input data and the
-
Building a Reading Companion App for Books with AI Context
By
–
Nice! I really want to build a reading companion app for books. E.g. I am re-reading LoTR again, you could imagine stuffing all of it (and discussion boards related commentary and chatter) into context and making it very easy to ask questions, clarifications, discussions. There's
-
nanoGPT for Education: Exploring Production-Ready Alternatives
By
–
btw nanoGPT is meant for education, possibly have a look at some of the slightly bit more "prod" repos i link to it in the readme, e.g. litgpt or tinyllama. When you look at the code it will look quite nanoGPT-like and recognizable, but possibly a bit more battle-tested.
-
Mathematics and Calculations in a Universe Without Physical Laws
By
–
(I think this is the right appeal, it doesn't appear like math science or engineering would be supported in such a Universe. Forget quantum physics etc., would even simple calculations like multiplying two numbers "work" and how wouldn't you always get hallucinations)
-
Simulation Hypothesis: Neural Approximation Over Physical Exactness
By
–
New simulation hypothesis drop.
Maybe the simulation is not physical and exact but neural and approximate.
i.e. not about simulating fields or particles with physical equations but a giant Diffusion Transformer++ creating a large "dream". -
Luma Labs AI Video Generation from Images Impresses Karpathy
By
–
wow. The new model from @LumaLabsAI extending images into videos is really something else. I understood intuitively that this would become possible very soon, but it's still something else to see it and think through future iterations of.
— Andrej Karpathy (@karpathy) 13 juin 2024
A few more examples around, e.g. the… https://t.co/zH1xjAgrNnwow. The new model from @LumaLabsAI extending images into videos is really something else. I understood intuitively that this would become possible very soon, but it's still something else to see it and think through future iterations of. A few more examples around, e.g. the
-
Scaling Risk: Why Small-Scale Proofs Don’t Guarantee Success
By
–
Two related good quotes I heard recently: "You can prove that something won't work at small scale, but not that something works at small scale" "There's way more ideas out there than compute that's willing to take a risk on it"