We wanted to share some more granular details about the Jamba 1.5 model family – and specific benchmarks on latency, context window, and quality. [1/6]
@ai21labs
-
Head-to-Head Latency Test Results Comparison
By
–
We ran a head-to-head #latency test, with the same hardware and same prompts.
— AI21 Labs (@AI21Labs) 22 août 2024
Want to guess who won? 🤔 pic.twitter.com/SJGTKk7SCLWe ran a head-to-head #latency test, with the same hardware and same prompts. Want to guess who won?
-
Jamba 1.5 Mini and Large Models Released on Hugging Face
By
–
Jamba 1.5 Mini and Large on Hugging Face: https://
huggingface.co/collections/ai
21labs/jamba-15-66c44befa474a917fcf55251
… -
AI21 Labs Makes Jamba Models Available on Together AI
By
–
Thanks @togethercompute – we're exciting to be making our Jamba models available on Together AI soon!
-
AI21 Labs Offers Course on Hybrid SSM-Transformer Architecture
By
–
Congrats Andrej! We'd love to help you create a course on building a hybrid SSM-Transformer architecture and all of the awesome benefits of that.
-
Jamba’s 256K Context Reveals fused_moe Kernel Issues
By
–
BTW, Since Jamba supports a 256K context with high throughput, we also stumbled upon an issue where the fused_moe kernel didn’t work well in long contexts. Others seems to have had this too, according to some other open issues
-
MoE Token Independence Enables Efficient Long Context Processing in vLLM
By
–
@avshalomm solved it by utilizing the fact that actually, there is no interaction between different tokens in the MoE block, so we can iterate over the long context in chunks. This was also merged and now fixed in vLLM
-
Jamba Model Now Supported in vLLM for Efficient Serving
By
–
Jamba support is now live on vLLM Due to its novel hybrid SSM-Transformer arch, Jamba didn’t work out-of-the-box in vLLM. our own @MorZusman worked together with @vllm_project to integrate Jamba for an efficient serving in vLLM
-
Jamba Modeling and Mamba Cache Management Updates
By
–
Things in this PR:
– Jamba modeling file, supporting Jamba variations
– Mamba cache management (also benefit other Mamba-based models)
– Send more request related properties to the forward pass, for model-specific implementations (including things needed for speculative decoding) -
AI Pals: Voice-Enabled Avatars Powered by Jamba’s Long Context
By
–
In 3rd: AI Pals The team built avatars that could communicate via voice-over. By leveraging Jamba’s long context, entire books were used within a single prompt, thereby decreasing latency. Built by Anto Patrex, Anand Vallamsetla, Justin Jung, & Lars Traaholt Vågnes