Interestingly, the watermark tokens can be embedded with negligible impact on text quality.
LLMS
-
Watermarking AI text to detect ChatGPT-generated content
By
–
"Watermarking" AI-generated text. Researchers from University of Maryland propose a way to discretely embed (and detect) special tokens in order to accurately determine whether or not text was generated from something like ChatGPT. ↓
-
Emergence as Framework for Understanding Language Model Scaling
By
–
Overall, emergence is nice framework for viewing language models
– Emergent abilities cannot be predicted via scaling plots for small models
– Keep scaling, and we might see more emergent abilities 🙂 See our TMLR 2022 piece (w/ survey certification): -
U-shaped Scaling: Model Performance Recovery in Larger Language Models
By
–
Another newer example of emergence that I’m excited about is U-shaped scaling, where even if model performance goes down from small → medium models, it can go back up for large models. See our paper: https://
x.com/_jasonwei/stat
us/1588605909781319680
… Inverse scaling benchmark: -
Instruction Tuning as an Emergent Ability in Large Language Models
By
–
One of the most interesting emergent abilities IMO is instruction tuning. Anthropic and Flan-LaMDA suggest that zero-shot performance can improve from RLHF and NLP benchmark instruction tuning (although text-davinci usually loses to code-davinci). https://
arxiv.org/abs/2204.05862 -
Emergent Abilities in Language Models Through Scaling
By
–
Yesterday I gave a lecture at @Stanford
's CS25 class on Transformers! The lecture was on how “emergent abilities” are unlocked by scaling up language models. Emergence is one of the most exciting phenomena in large LMs… Slides: -
Emergent Abilities in Large Language Models Over Past Year
By
–
Throughout the past year, there have been hundreds of emergent abilities, which can only be observed in large-enough language models. I previously made a list of them (more than 100):https://t.co/QLfwNUeIBx
— Jason Wei (@_jasonwei) 25 janvier 2023Throughout the past year, there have been hundreds of emergent abilities, which can only be observed in large-enough language models. I previously made a list of them (more than 100):
-
GPT as Backend: LLM as English Interpreter for Development
By
–
"GPT is all you need for backend". This was the most inspirational project from the hackathon over the weekend, hard to stop thinking about. LLM is a kind of equivalent of the Python interpreter, except it interprets English, and has knowledge and common sense.
-
Meta Withdraws Galactica LLM After Trolling and Bias Issues
By
–
Correction: Meta released their "Galactica" LLM but withdrew it 3 days after because it was trolled and full of bias
-
Concerns About Irresponsible Release of Large Language Models
By
–
Sharing with everyone what I have been repeating to media and interested parties since the last months: it was irresponsible to release a large language model that still has severe limitations – and there are others even superior to it #ChatGPT #GPAI @GPAI_PMIA @OECDEduSkills