Summary of stable FP8 training recipe for Llama models:
1. Clip learning rate when 2nd moment estimator is outdated by coming spike – see https://
arxiv.org/abs/2304.13013
2. SmoothQuant: smooths activation outliers by migrating the quantization difficulty to weights – see
@thom_wolf
-
Stable FP8 Training Recipe for Llama Models
By
–
-
SmolLM 360M: Instant Lightweight Language Model Browser Demo
By
–
Sharing one demo which is blowing my mind, the new instant SmolLM 360M running live in your browser: https://
huggingface.co/spaces/Hugging
FaceTB/instant-smollm
… And find more info here: https://
huggingface.co/collections/Hu
ggingFaceTB/smollm-6695016cad7167254ce15966
… And here: -
Synthetic Data and Small Language Models Journey
By
–
It’s Sunday morning we have some time with the coffee so let me tell you about some of our recent surprising journey in synthetic data and small language models. This post is prompted by the coming release of an instant, in-browser model called SmolLM360 (link at the end) The
-
Modular Speech-to-Speech Pipeline Achieves Sub-500ms Latency
By
–
this modular speech-to-speech pipeline repo with <500 ms second latency (on a local GPU) is going crazy
— Thomas Wolf (@Thom_Wolf) 16 août 2024
Impressive what a well tuned pipeline of stt-llm-tts can do
cool work by @eustachelb with @reach_vb and @andi_marafioti
find the code here: https://t.co/HrbqSWQ3sj pic.twitter.com/HDkEnVntKNthis modular speech-to-speech pipeline repo with <500 ms second latency (on a local GPU) is going crazy Impressive what a well tuned pipeline of stt-llm-tts can do cool work by @eustachelb with @reach_vb and @andi_marafioti find the code here: https://
github.com/eustlb/speech-
to-speech
… -
Open-Source OCR Development for Asian Languages
By
–
Who’s working on (open-source ai for) OCR for languages spoken in Asia? Thinking in particular Hindi, Nepali but also wider
-
FLUX Image Generation Model Creates Open-Source AI Storm
By
–
wow FLUX the image-generation model from @bfl_ml has taken the open-source ai world totally by storm never seen so many derivative/spaces/demos of a model trending at the same time
-
New Model Tree Navigation Feature Launched on Hub
By
–
Have you noticed the new "model tree" section we just added on the Hub? You can use it to navigate the tree of models finetuned, merged, adapted, quantized from each other! I find it really cool and with over a million models on the hub it's a really interesting way to move
-
Idefics3: State-of-the-Art Vision+Text Model Released
By
–
Llama 3.1 multimodal is not released yet? We got you covered with the new Idefics3 a state of the art Vision+Text model based on the latest Llama 3.1
-
Open-source AI models: transparency over performance gains
By
–
Open-source AI models are like sport without doping Performances can be slightly lower but in the end, transparency is more sustainable, healthy and exciting to watch and participate in 🙂
-
Small, Local and Highly Efficient LLMs
By
–
Small, local and highly efficient LLMs https://t.co/5pQoIKInL8
— Thomas Wolf (@Thom_Wolf) 29 juillet 2024Small, local and highly efficient LLMs