On code generation benchmarks, LLaMA-62B outperforms cont-PaLM (62B) as well as PaLM-540B.
@guillaumelample
-
LLaMA-65B Outperforms Chinchilla and PaLM on Reasoning Benchmarks
By
–
On Common Sense Reasoning, Closed-book Question Answering, and Reading Comprehension, LLaMA-65B outperforms Chinchilla 70B and PaLM 540B on almost all benchmarks.
4/n -
LLaMA-65B Outperforms Minerva-62B on GSM8k Without Mathematical Fine-tuning
By
–
LLaMA-65B outperforms Minerva-62B on GSM8k, even though it has not been fine-tuned on any mathematical dataset. On the MATH benchmark, it outperforms PaLM-62B (but is quite below Minerva-62B)
5/n -
Models trained on 1T tokens with continued improvement at 7B scale
By
–
All our models were trained on at least 1T tokens, much more than what is typically used at this scale.
Interestingly, even after 1T tokens the 7B model was still improving.
3/n -
Open Dataset Approach: LLaMA’s Reproducible Alternative to Chinchilla and GPT-3
By
–
Unlike Chinchilla, PaLM, or GPT-3, we only use datasets publicly available, making our work compatible with open-sourcing and reproducible, while most existing models rely on data which is either not publicly available or undocumented.
2/n -
Meta Releases LLaMA Foundation Models Open Source
By
–
Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters.
LLaMA-13B outperforms OPT and GPT-3 175B on most benchmarks. LLaMA-65B is competitive with Chinchilla 70B and PaLM 540B.
The weights for all models are open and available at https://
research.facebook.com/publications/l
lama-open-and-efficient-foundation-language-models/
…
1/n -
Pre-training with Feedback: Computational Cost Concerns
By
–
This would be sad if pre-training with feedback is actually better, because pre-training is by far the most expensive part of the training, and you wouldn't want to re-train from scratch every time you update your reward model.
-
Pre-training scale reduces need for human feedback fine-tuning
By
–
Haven't read carefully, but Figure 1 suggests that with more pre-training, you need less fine-tuning with human feedback (which is usually the case with fine-tuning). So at scale this would make potentially no difference?
-
Recent papers on LLMs for formal mathematics and theorem proving
By
–
We had two papers on this topic recently, one using a MCTS approach on formal systems (such as Lean or Metamath): https://
arxiv.org/abs/2205.11491 and one where we combine LLMs with formal provers to verify the generations: https://
arxiv.org/abs/2210.12283 -
Pre-trained Language Models Essential for Agent Communication
By
–
If AZ cannot rely on a pre-trained LM to communicate, it is impossible (there is no reason the model learns with self play to communicate with valid sentences). Other issue will of course be the infinite action space (which we also had in theorem proving) but this is manageable