I'd be interested too but I don't expect much. Still a cool approach, with lots of potential gains in scaling more efficiently (training compute =>> inference compute)
@maximelabonne
-
Tele-FLM-1T: Progressive Training and Depth Growth Techniques
By
–
Here are the juicy lessons from the 1T model Tele-FLM-1T: – Trained progressively in 3 stages: 52B -> 102B -> 1T parameters on ~2T tokens – Depth growth technique: selects layers to duplicate based on their input-output distance metrics, prioritizing middle layers with smaller
-
Learning Orthogonalized Matrices vs Computing Refusal Directions
By
–
I'd say it's different because you learn the orthogonalized matrix with OFT while you compute (you don't learn) your refusal direction here
-
Model Weight Redaction Techniques in AI Systems
By
–
Yes it's kind of redacting the model's weights in a way
-
Llama 3.0 Previous Checkpoint Release Details
By
–
It's the previous checkpoint they presented when Llama 3.0 was released
-
MMLU Pro benchmark evaluation data analysis
By
–
Agreed, MMLU Pro is not perfect either but we have enough data points to make a plot now
-
DeepSeek-V2 and Mistral Large 2 Added to Updated Figure
By
–
Due to popular demand, I've updated this figure to include DeepSeek-V2 and Mistral Large 2. It's also more zoomed for readability.
-
Llama 3.0 Update: Libraries Not Current, Long Context Impact
By
–
The libraries are not up-to-date yet, so it received the Llama 3.0 treatment. It's a good question, long context might have suffered depending on how it's handled, but I haven't tested it.
-
First Uncensored Llama 3.1 Instruct Model Released
By
–
I'm releasing the first uncensored Llama 3.1 Instruct model with GGUF quants This is an abliterated model that works well in my tests. You can learn more about it in my article. Model: https://
huggingface.co/mlabonne/Meta-
Llama-3.1-8B-Instruct-abliterated
… Article: https://
huggingface.co/blog/mlabonne/
abliteration
…