Fine-tune with Spectrum Dolphin models use this technique to target the most important layers to fine-tune, like in their recent Qwen2-72B version https://
huggingface.co/cognitivecompu
tations/dolphin-2.9.2-qwen2-72b
… You can use their repo to identify the layers and the "unfrozen_parameters" key in your Axolotl config.
OPEN SOURCE
-
Fine-tuning LLMs with Spectrum: Targeting Key Layers Efficiently
By
–
-
ARC Prize: $1M+ Competition for Open AGI Progress
By
–
ARC PRIZE: A $1M+ competition towards open AGI progress. Exciting initiative hosted by @mikeknoop @fchollet https://
arcprize.org/blog/launch -
Llama-3 vs Qwen-2: Local LLM Comparison Guide
By
–
Compare Llama-3 and Qwen-2 running locally on your computer (100% free and without internet) pic.twitter.com/TOBPHNEiZm
— Shubham Saboo (@Saboo_Shubham_) 12 juin 2024Compare Llama-3 and Qwen-2 running locally on your computer (100% free and without internet)
-
Llama 3 Chat Template Optimization for Fine-tuning
By
–
Thanks actually I realised that Axolotl had a Llama 3 chat template after this finetune. It'd be better to use it instead of chatml for sure. It would be better but tbh I've milked this recipe a bit too much 🙂
-
Phi-3 MoE Implementation Discussion
By
–
I haven't checked but that sounds good indeed, you might be able to directly do a phi-3 MoE with this
-
Open-Source Agentic Machine Translation Demonstration Released
By
–
I think AI agentic machine translation has huge potential for improving over traditional neural machine translation, and am releasing as open-source a demonstration I'd been playing with as a fun weekend project. Using an agentic workflow, this demonstration (i) Prompts an LLM
-
Deploy GPT-4 Performance to Snowflake with Open Source Models
By
–
Bring #GPT4 performance to your @SnowflakeDB #DataCloud without spending 10s of thousands of dollars a month And, you can do this with #opensource models—no need to share data w/ a commercial #LLM provider or defer model ownership Save your spot: https://
pbase.ai/3yZ69qx -
Create Your Phi MoE: Phixtral Reproduction Guide
By
–
Create your Phi MoE In January, I released Phixtral, the first MoE based on Phi-2. I said I would release the code, but Microsoft decided to modify Phi-2, which broke my version. Thanks to Vishal Rajendra Kumar, we now have a way to reproduce Phixtral with his Phi2xtral
-
AutoTrain Integrates Unsloth for Fast LLM Fine-tuning
By
–
AutoTrain + Unsloth = AutoTrain has now added support for unsloth which means you can use unsloth's optimizations to finetune LLMs super-fast and with much less memory And all you need to do is set unsloth parameter to true P.S. You can use the unsloth param in cli