“Wait, code-davinci-002 is the most capable GPT-3 model for natural language as well as code, on top of having a context window twice as big??” “Always has been.”
@goodside
-
Performance in ML means accuracy, not speed
By
–
(For non-ML folks: “performance” means being right, not being fast — accuracy, F1 score, etc.)
-
Instruction prompting: powerful, but with performance and style costs.
By
–
Instruction prompting is wonderful. Easier than k-shot for nearly every task, and tackles some tasks (e.g. applying large lookup/policy tables) impossible to specify fully through demonstration. But it isn’t free. You pay a tax in performance — and, I’d add, in style.
-
ChatGPT sacrifices in-context learning for dialog context modeling
By
–
“ChatGPT seems to have sacrificed nearly all of its in-context learning ability to trade for the ability to model dialog context.”
-
text-davinci-003 recovers in-context learning ability but worse than code-davinci-002
By
–
“text-davinci-003 recovered (but still worse than code-davinci-002) some in-context learning ability that is lost in text-davinci-002 (presumably because it tunes the model with LM mix-in) and further improved zero-shot ability (thanks to RLHF).”
-
text-davinci-002 instruction tuning reduces in-context learning, boosts zero-shot ability
By
–
“text-davinci-002, released in May-Jun 2022, is a supervised instruction-tuned model based on code-davinci-002. It is very likely that the instruction tuning on text-davinci-002 decreased the model’s in-context learning ability but increased the model’s zero-shot ability […]”
-
Code-davinci-002 is the most capable GPT-3.5 variant for natural language
By
–
So many great insights here:
“Although called Codex, code-davinci-002 is probably the most capable GPT-3.5 variant for natural language (better than text-davinci-002 and 003). It is very likely code-davinci-002 is trained on both text and code, then tuned on instructions […]” -
Wishing for editing responses in task-oriented chat UI
By
–
Top of wish-list: Editing the responses. For task-oriented chat, prior task requests serve (elegantly!) as k-shot examples. But the UI prevents corrections except via token-wasting repetition or prose feedback. Playground is still a better feedback/correction loop for me.
-
History of GPT-3 and its emergent abilities explained
By
–
You think ChatGPT is amazing — you’ve been hacking on computers for years, but this you can’t explain. How did we get here, and so suddenly? How does it know and do so much? @Francis_YAO_ of @EdinburghNLP explains the history of GPT-3: https://
yaofu.notion.site/How-does-GPT-O
btain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
… -
ChatGPT praised for low confabulation and dialog safety
By
–
Don’t want to diminish what an achievement ChatGPT is. Dialog applications matter and need stricter safety standards. ChatGPT is by far the least confabulatory of any LLM I’ve used, in a way I have no idea how to achieve otherwise. It is good at what it does.