When we started this project, our goal was to reproduce Chinchilla, so we did not move too much away from the original architecture. Retrospectively, we should probably have done what PALM did: multi query + increase FeedForward hidden size to make up for the loss of parameters
@guillaumelample
-
LLaMA 65B Now Runs on MacBook Pro
By
–
LLaMA 65B can run on a MacBook! With a different model architecture it could probably run quite faster (we didn't use multi query, for instance) https://t.co/SMdlZC5NOo
— Guillaume Lample @ NeurIPS 2024 (@GuillaumeLample) 11 mars 2023LLaMA 65B can run on a MacBook! With a different model architecture it could probably run quite faster (we didn't use multi query, for instance)
-
Weights Only Distribution Without Optimizer States
By
–
We only provide the weights, no optimizer states.
-
Environmental Impact of Training Data Centers Simulation
By
–
We mention "in the same data center" twice in the caption so it is clear that this is a simulation. The table aims at informing people on what kind of training is more environmental friendly (to inform future training decisions, for instance), and this is precisely what it does.
-
Emissions comparison between BLOOM and LLaMA training
By
–
Well, in the paper we are talking about a BLOOM and a LLaMA. This is the question answered by the table: how much emissions will this training emit compared to this one? The caption very clearly states "in the same data center" in bold, and is factually correct information.
-
Comparing Model Emissions Across Different Countries Misleading
By
–
No. Comparing the emissions of different models trained in different countries is what would actually be misleading. In this case it would falsely suggest that training a BLOOM model emits less CO2 than training a LLaMA, while this is the opposite.
-
Loss spikes during training were rare and brief
By
–
We indeed had a handful of loss spikes, but these were very rare (maybe less than 5 or 10 over the entire training) and never lasted more than a couple of iterations, so we didn't have to do anything like skipping batches or lowering learning rate.
-
Model trained with 2048 token sequence length
By
–
It was trained with a sequence length of 2048 tokens.
-
Dataset Contamination Impact in PaLM Models Appears Minimal
By
–
Looking at the "Dataset Contamination" section in PaLM, it does not seem to make a big difference does it? Which makes sense, would be surprising if it does after data deduplication +single iteration over the training set (also our common crawl data is <= 2020)
-
Instruction Finetuning Results: LLaMA-I Outperforms Flan-PaLM
By
–
We also briefly tried instruction finetuning using the approach of Chung et al. (2022).
The resulting model, LLaMA-I, outperforms Flan-PaLM-cont (62B) on MMLU and showcases some interesting instruct capabilities.
7/n