“ChatGPT seems to have sacrificed nearly all of its in-context learning ability to trade for the ability to model dialog context.”
MACHINE LEARNING
-
Instruction prompting: powerful, but with performance and style costs.
By
–
Instruction prompting is wonderful. Easier than k-shot for nearly every task, and tackles some tasks (e.g. applying large lookup/policy tables) impossible to specify fully through demonstration. But it isn’t free. You pay a tax in performance — and, I’d add, in style.
-
text-davinci-002 instruction tuning reduces in-context learning, boosts zero-shot ability
By
–
“text-davinci-002, released in May-Jun 2022, is a supervised instruction-tuned model based on code-davinci-002. It is very likely that the instruction tuning on text-davinci-002 decreased the model’s in-context learning ability but increased the model’s zero-shot ability […]”
-
text-davinci-003 recovers in-context learning ability but worse than code-davinci-002
By
–
“text-davinci-003 recovered (but still worse than code-davinci-002) some in-context learning ability that is lost in text-davinci-002 (presumably because it tunes the model with LM mix-in) and further improved zero-shot ability (thanks to RLHF).”
-
Code-davinci-002 is the most capable GPT-3.5 variant for natural language
By
–
So many great insights here:
“Although called Codex, code-davinci-002 is probably the most capable GPT-3.5 variant for natural language (better than text-davinci-002 and 003). It is very likely code-davinci-002 is trained on both text and code, then tuned on instructions […]” -
AI Generates Code from Text
By
–
AI models that take sentences, comments, or statements and build the corresponding piece of code for you. So, what to make of it? 3/7
-
History of GPT-3 and its emergent abilities explained
By
–
You think ChatGPT is amazing — you’ve been hacking on computers for years, but this you can’t explain. How did we get here, and so suddenly? How does it know and do so much? @Francis_YAO_ of @EdinburghNLP explains the history of GPT-3: https://
yaofu.notion.site/How-does-GPT-O
btain-its-Ability-Tracing-Emergent-Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
… -
Meeting Ranjani in Person After Year and Half Online
By
–
Met @ranjani_mani in person after knowing her for the past year and a half! Amazed, energized and inspired by what she does. I guess LinkedIn is the modern day platform for the (RIP) Pen-pals! Yes, if you got the NLP or ML Engineering chops and want to…
https://
lnkd.in/d2yuMkUS -
Neural Operators Learn Function Mappings for Zero-Shot High-Resolution Predictions
By
–
Neural operators provide evaluations outside the training grid and make higher-resolution predictions in a zero-shot manner. This is because neural operators learn mappings between function spaces, so they learn output as a function, not a vector https://
arxiv.org/pdf/2108.08481
.pdf
… -
Face Recognition Model Quality Issues Pre-Tuning
By
–
Pic example of how bad it is at faces pre-tuning