Very excited to release our second model, Mixtral 8x7B, an open weight mixture of experts model.
Mixtral matches or outperforms Llama 2 70B and GPT3.5 on most benchmarks, and has the inference speed of a 12B dense model. It supports a context length of 32k tokens. (1/n)
@guillaumelample
-
Mixtral 8x7B: Open Weight Mixture of Experts Model Released
By
–
-
Guillaume Lample at NeurIPS 2023 discussing open source and Mistral AI
By
–
I will be at #NeurIPS2023 this week. Feel free to reach out if you want to talk about open source or if you want to know more about @MistralAI (also we are hiring!)
-
Mistral AI team at ICCV Paris conference
By
–
We will be at ICCV in Paris for the next couple of days. Feel free to reach out if you are interested about @MistralAI !
-
Thanks to compute providers and partners for model release support
By
–
Huge thanks to our compute providers @CoreWeave and the EuroHPC, @tri_dao and @d_haziza for their help with FlashAttention and xFormers, and @huggingface
, vLLM, @skypilot_org
, FastChat for their help and support with this release. -
Mistral AI team builds advanced MLOps stack in three months
By
–
Very proud of the Mistral AI team who rebuilt a top-performance MLops stack, and designed a very sophisticated data processing pipeline, from scratch, in less than 3 months.
-
GQA and Sliding Window Optimization for Efficient Decoding
By
–
We trained it with GQA and a sliding window of 4096 tokens, resulting in constant cache size and a linear decoding speed. Our changes to FlashAttention v2 and xFormers to support sliding window are available to the community.
-
Mistral 7B Released: Outperforms Llama 2 13B on All Benchmarks
By
–
Mistral 7B is out. It outperforms Llama 2 13B on every benchmark we tried. It is also superior to LLaMA 1 34B in code, math, and reasoning, and is released under the Apache 2.0 licence. https://
mistral.ai/news/announcin
g-mistral-7b/
… -
Fake Mistral AI Account Alert: Report and Unfollow
By
–
Somebody created a fake account "ai_mistral" and started following ML people. It has now 2k+ followers, but we have no idea who this.. If you follow it, please unfollow & report it !
-
Guillaume Lample leaves Meta to launch Mistral.AI startup
By
–
Life update: I recently left Meta, and we are starting http://
Mistral.AI, a new AI company with @arthurmensch and @tlacroix6 -
Default Settings Achieve Strong Results Without Advanced Techniques
By
–
"No fancy prompting engineering, no fancy decoding, everything by default."