Random quick note on Transformer block unification. People are usually a bit surprised that the MLP and Attention blocks that repeat in a Transformer can be re-formated to look very similar, likely unifiable. The MLP block just attends over data-independent {key: value} nodes:
@karpathy
-
Batchnorm as a Challenging Interview Question for Candidates
By
–
For a while I thought an excellent interview question would be to ask the candidate about batchnorm and measure the amount to which their face contorts
-
GPT as Backend: LLM as English Interpreter for Development
By
–
"GPT is all you need for backend". This was the most inspirational project from the hackathon over the weekend, hard to stop thinking about. LLM is a kind of equivalent of the Python interpreter, except it interprets English, and has knowledge and common sense.
-
English Becomes the Hottest New Programming Language
By
–
The hottest new programming language is English
-
Organizations of LLMs Collaborating Like Human Teams
By
–
yeah, I was thinking about how we could eventually have orgs of LLMs just like orgs of people, performing more complex tasks. Useful because just like people they could specialize, execute in parallel, hold meetings. Maybe that "org code" is written in something like langchain.
-
Large Transformer Model Inference Optimization Techniques Overview
By
–
Excellent overview/pointers for "Large Transformer Model Inference Optimization" techniques (and blog more generally).
-
Cross-attention in Transformer architecture article recommendation
By
–
nice article on cross-attention as supplementary too https://
vaclavkosar.com/ml/cross-atten
tion-in-transformer-architecture
… -
Clarification on Query and Key in Attention Mechanisms
By
–
nice, exactly! 🙂
(except you're swapping q's and k's – q is the query, the "what am i looking for", k is the key, the "what do i have", and in encoder-decoder the key,value from come from side. admittedly confusing because in dictionaries the _key_ is the "lookup" information.) -
Comprehensive web-based UI for side-by-side model comparison
By
–
great! would be neat to have a more comprehensive web-based comparison UI that has a number of categories of tasks with the two models side by side with metrics, and when you click you get the "proof" behind the aggregate metric, with underlying examples and judgements etc.
-
Karpathy Recommends Lambda API for Easy On-Demand GPU Access
By
–
I like and use @LambdaAPI cloud GPUs, I think the easiest way to spin up an on-demand GPU instance that I'm currently aware of.