2/ These two [1] https://
arxiv.org/abs/2205.11916 , [2] https://
arxiv.org/abs/2211.01910 are good examples that the prompt can further program the "solution strategy", and with a good enough design of it, a lot more complex multi-step reasoning tasks become possible.
@karpathy
-
Prompts Enable Complex Multi-Step Reasoning Through Strategic Design
By
–
-
LLMs In-Context Learning and Prompt Programming Capabilities
By
–
This tweet went wide, thought I'd post some of the recent supporting articles that inspired it.
1/ GPT-3 paper showed that LLMs perform in-context learning, and can be "programmed" inside the prompt with input:output examples to perform diverse tasks https://
arxiv.org/abs/2005.14165 -
Move Test Time Compute to Training Time When Possible
By
–
it's good except as a rule of thumb you always want to move test time compute into train time compute, to whatever extent possible.
-
Content-Conditioned Q&A Assistants: A Future Feature
By
–
content-conditioned Q&A assistant is a prominent feature of the future.
-
Step-by-Step Thinking Needs Correct Answers in GPT Models
By
–
One of my favorite results in 2022 was that it's not enough to just think step by step. You must also make sure to get the right answer 😀 https://
sites.google.com/view/automatic
-prompt-engineer
…
(actually a nice insight into a psychology of a GPT; it pays to condition on a high reward) -
Andrej Karpathy Rejoins OpenAI to Build AI Future
By
–
Some personal news: I am joining OpenAI (again :)). Like many others both in/out of AI, I am very inspired by the impact of their work and I have personally benefited greatly from it. The future potential is especially exciting; it is a great pleasure to jump back in and build!
-
Single Flag Changes Can Match Months of Optimization Work
By
–
I love how sometimes changing one integer/flag can have the same impact as a 1 month optimization project. You just know there is some OMP_NEVER_HEARD_OF=3 that gets addition 3% MFU. Or my personal favorite – that undocumented bios flag that only 4 people on Earth know exists 😀
-
Linear and LayerNorm Biases Appear Useless in Experiments
By
–
I'll give it a shot! Btw it is biases in both Linear and LayerNorm that appear to be useless (from my admittedly smaller scale experiments).
-
Vocabulary Size Optimization: 25% Speed Boost Through Kernel Alignment
By
–
The most dramatic optimization to nanoGPT so far (~25% speedup) is to simply increase vocab size from 50257 to 50304 (nearest multiple of 64). This calculates added useless dimensions but goes down a different kernel path with much higher occupancy. Careful with your Powers of 2.