Base LLMs (non-finetuned) make very strong few-shot classifiers. Describe task in English, give few examples, read off the label probabilities on test example. No gradient-based optimization necessary. It brings a cannon to a knife fight but is fast, convenient, strong baseline.
@karpathy
-
MIT Licensed Evaluations and Benchmark Releases on GitHub
By
–
I didn’t work on this project personally but I feel like “undermining” is a strong word. Did you feel the same way for eg BIG-bench / HELM releases? Do you think it is good that there are more MIT licensed evals on GitHub?
-
OpenAI Open Sources Evals Framework for GPT-4 Model Evaluation
By
–
Less publicized but highly awesome aspect of GPT-4 launch was that OpenAI open sourced an evals framework, allowing us to crowdsource model evaluations at scale . The repo is getting some very high quality PRs (rewarded with GPT-4 access). I <3 evals; `pip install evals` https://
x.com/kondrich2/stat
/kondrich2/status/1635687409668227072
… -
GPT-4 livestream showcases unprecedented new capabilities for developers
By
–
The GPT-4 developer livestream (
https://
youtube.com/watch?v=outcGt
bnMuQ
…) was a great preview of new capability. Not sure I can think of a time where there was this much unexplored territory with this much new capability in the hands of this many users/developers. -
Vision Capability Rollout Excitement and Uncertainty
By
–
Plot twist: it's solved or probably it's not solved or we're not sure . Really looking forward the vision capability rolling out publicly soon, unlocks a ton of new/exciting uses.
-
Vision Capability Breakthrough: Testing Confirms Strong Performance
By
–
We tried and it solves it :O. The vision capability is very strong but I still didn't believe it could be true. The waters are muddied some by a fear that my original post (or derivative work there of) is part of the training set. More on it later.
-
GPT-4 Released: Multimodal AI Model Now Available on ChatGPT Plus
By
–
🎉 GPT-4 is out!!
— Andrej Karpathy (@karpathy) 14 mars 2023
– 📈 it is incredible
– 👀 it is multimodal (can see)
– 😮 it is on trend w.r.t. scaling laws
– 🔥 it is deployed on ChatGPT Plus: https://t.co/WptpLYHSCO
– 📺 watch the developer demo livestream at 1pm: https://t.co/drEkxQMC9H https://t.co/WUYzwyxOqaGPT-4 is out!!
– it is incredible
– it is multimodal (can see) – it is on trend w.r.t. scaling laws
– it is deployed on ChatGPT Plus: http://
chat.openai.com
– watch the developer demo livestream at 1pm: https://
youtube.com/live/outcGtbnM
uQ?feature=share
… -
The Transformer as Its Own Attacker: An Ironic Meta
By
–
not bad except the meta is that the attacker is the Transformer itself
-
Dropout=1.0 Issue: PyTorch Should Raise ValueError
By
–
The model is not "turned off during training". With dropout=1.0, for dropout layers you'll get all zero at train and, apparently, identity at test. I don't think pytorch should have allowed dropout=1.0. It should be ValueError, not sure I get the reasoning there.
-
Dropout Layers Leak Training Phase Information in Transformers
By
–
Dropout layers in a Transformer leak the phase bit (train/eval) – small example. So an LLM may be able to determine if it is being trained and if backward pass follows. Clear intuitively but good to see, and interesting to think through repercussions of