We talk about residual streams and how weight orthogonalization works. See my previous thread for more info about NeuralDaredevil-8B
@maximelabonne
-
Abliteration Technique Uncensors LLMs Like NeuralDaredevil-8B
By
–
Uncensor any LLM with abliteration I wrote an article on @huggingface about abliteration and how NeuralDaredevil-8B was created. It's a cool technique with a lot of potential. Special thanks to @failspy for proofreading the article. Article: https://
huggingface.co/blog/mlabonne/
abliteration
… -
Merge Results Expected at 90 Percent Performance
By
–
It would be interesting to see the results. I expect this merge to be like 90% as good as the full merge
-
AutoQuant: Automatically Quantize LLMs in Colab
By
–
AutoQuant: automatically quantize your LLMs in Colab Following llama.cpp's updates to fix Llama 3 quants, it's time to communicate again about AutoQuant It's a user-friendly Colab to create your own GGUF, EXL2, AWQ, and HQQ quants Colab: https://
colab.research.google.com/drive/1b6nqC7U
ZVt8bx4MksX7s656GXPM-eWw4?usp=sharing
… -
Mini Model Surpasses Mixtral on MMLU Benchmark Score
By
–
Soooo what do you think about this MMLU score? Specifically for the Mini model, which is higher (68.8) than Mixtral's (68.4).
-
FP32 vs BF16 Format Issue in Model Training
By
–
Ah actually I didn't notice. It's in FP32 instead of BF16. It's fine but that wasn't the plan
-
Model Weights FP16 Specifications and Technical Requirements
By
–
Where do you see that? The weights should be about 16 GB in FP16
-
FIM Performance with Instruct Models Unexplored
By
–
I've never tried FIM with an instruct model to be honest, I didn't expect it to work well. Maybe they all do and I didn't know?
-
Open-source community under attack from alleged psyops campaign
By
–
Insane psyops from @osanseviero to kill the open-source community's productivity
-
GPT-4o vs Claude Opus: Code Generation Performance Comparison
By
–
Not sure they're doing the exact same eval, but GPT-4o reports 90.2% vs. 84.9% for Claude 3 Opus on HumanEval (
https://
openai.com/index/hello-gp
t-4o/
…). I don't expect Codestral 22B to be on par, but it can do FIM