“ChatGPT seems to have sacrificed nearly all of its in-context learning ability to trade for the ability to model dialog context.”
LLMS
-
Instruction prompting: powerful, but with performance and style costs.
By
–
Instruction prompting is wonderful. Easier than k-shot for nearly every task, and tackles some tasks (e.g. applying large lookup/policy tables) impossible to specify fully through demonstration. But it isn’t free. You pay a tax in performance — and, I’d add, in style.
-
text-davinci-002 instruction tuning reduces in-context learning, boosts zero-shot ability
By
–
“text-davinci-002, released in May-Jun 2022, is a supervised instruction-tuned model based on code-davinci-002. It is very likely that the instruction tuning on text-davinci-002 decreased the model’s in-context learning ability but increased the model’s zero-shot ability […]”
-
text-davinci-003 recovers in-context learning ability but worse than code-davinci-002
By
–
“text-davinci-003 recovered (but still worse than code-davinci-002) some in-context learning ability that is lost in text-davinci-002 (presumably because it tunes the model with LM mix-in) and further improved zero-shot ability (thanks to RLHF).”
-
Code-davinci-002 is the most capable GPT-3.5 variant for natural language
By
–
So many great insights here:
“Although called Codex, code-davinci-002 is probably the most capable GPT-3.5 variant for natural language (better than text-davinci-002 and 003). It is very likely code-davinci-002 is trained on both text and code, then tuned on instructions […]” -
Wishing for editing responses in task-oriented chat UI
By
–
Top of wish-list: Editing the responses. For task-oriented chat, prior task requests serve (elegantly!) as k-shot examples. But the UI prevents corrections except via token-wasting repetition or prose feedback. Playground is still a better feedback/correction loop for me.
-
History of GPT-3 and its emergent abilities explained
By
–
You think ChatGPT is amazing — you’ve been hacking on computers for years, but this you can’t explain. How did we get here, and so suddenly? How does it know and do so much? @Francis_YAO_ of @EdinburghNLP explains the history of GPT-3: https://
yaofu.notion.site/How-does-GPT-O
btain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
… -
ChatGPT Prompt Dataset Released in Datasets Library
By
–
ChatGPT prompt dataset just landed in the datasets lib
-
GPT-3 Training Data: C4 Dataset Technical Details
By
–
(Except that GPT3 is not trained on C4 per se @Francis_YAO_ but it’s a small tweak)
-
ChatGPT Reveals the Emptiness of Most Written Content
By
–
ChatGPT exposes how vacuous most of our writing is.