yeah, I was thinking about how we could eventually have orgs of LLMs just like orgs of people, performing more complex tasks. Useful because just like people they could specialize, execute in parallel, hold meetings. Maybe that "org code" is written in something like langchain.
@karpathy
-
Large Transformer Model Inference Optimization Techniques Overview
By
–
Excellent overview/pointers for "Large Transformer Model Inference Optimization" techniques (and blog more generally).
-
Cross-attention in Transformer architecture article recommendation
By
–
nice article on cross-attention as supplementary too https://
vaclavkosar.com/ml/cross-atten
tion-in-transformer-architecture
… -
Clarification on Query and Key in Attention Mechanisms
By
–
nice, exactly! 🙂
(except you're swapping q's and k's – q is the query, the "what am i looking for", k is the key, the "what do i have", and in encoder-decoder the key,value from come from side. admittedly confusing because in dictionaries the _key_ is the "lookup" information.) -
Comprehensive web-based UI for side-by-side model comparison
By
–
great! would be neat to have a more comprehensive web-based comparison UI that has a number of categories of tasks with the two models side by side with metrics, and when you click you get the "proof" behind the aggregate metric, with underlying examples and judgements etc.
-
Karpathy Recommends Lambda API for Easy On-Demand GPU Access
By
–
I like and use @LambdaAPI cloud GPUs, I think the easiest way to spin up an on-demand GPU instance that I'm currently aware of.
-
Training a 10M Parameter GPT Model on Shakespeare in 15 Minutes
By
–
We get a ~10M parameter model trained for about 15 minutes on 1 GPU on all of Shakespeare concatenated into one 1MB file. We then sample infinite fake Shakespeare from our baby GPT. Can you spot which one is real? At only 10M params on 1M characters, from-scratch, I hope so 🙂
-
Building Transformers: Self-Attention, Training, and GPT-3 Comparison
By
–
The second ~1hr builds up the Transformer: multi-headed self-attention, MLP, residual connections, layernorms. Then we train one and compare it to OpenAI's GPT-3 (spoiler: ours is around ~10K – 1M times smaller but the ~same neural net) and ChatGPT (i.e. ours is pretraining only)
-
Transformer Attention Mechanism: Baseline Model and Message Passing Introduction
By
–
First ~1 hour is 1) establishing a baseline (bigram) language model, and 2) introducing the core "attention" mechanism at the heart of the Transformer as a kind of communication / message passing between nodes in a directed graph.
-
Computer CoPilot Vision Now Tractable Through Text Translation
By
–
Computer CoPilot. Was very much the vision with OpenAI Universe https://
openai.com/blog/universe/ , though it was too early. Now feels tractable if you translate everything to/from text (e.g. like in WebGPT). Could be built e.g. as an extension of natbot https://
github.com/nat/natbot