Jamba-1.5 whitepaper is out!
The whitepaper details the architecture, training schemes, novelties and in-depth evaluations of our new long context hybrid SSM-Transformer models – Jamba-1.5-Large and Jamba-1.5-Mini. Arxiv: https://
arxiv.org/abs/2408.12570 Here are some highlights and
OPEN SOURCE
-
Jamba-1.5 Whitepaper Released: Hybrid SSM-Transformer Models
By
–
-
LM Studio v0.3 Released with Improved User Interface
By
–
Also enjoying the v0.3 of @LMStudioAI
, the new UI is slick -
Open-Source LLMs: Unsung Pioneer in AI Evolution
By
–
Doesn't get enough credit but IMO paved the way for open-source LLMs!
-
Stable FP8 Training Recipe for Llama Models
By
–
Summary of stable FP8 training recipe for Llama models:
1. Clip learning rate when 2nd moment estimator is outdated by coming spike – see https://
arxiv.org/abs/2304.13013
2. SmoothQuant: smooths activation outliers by migrating the quantization difficulty to weights – see -
Open Source Strategy: Commercial Failure or Planned Decision?
By
–
Was open source always the goal, or did sales not go well?
-
Scalable Autoregressive Image Generation with Mamba Architecture
By
–
Scalable Autoregressive Image Generation with Mamba discuss: https://
huggingface.co/papers/2408.12
245
… model: https://
huggingface.co/hp-l33/aim We introduce AiM, an autoregressive (AR) image generative model based on Mamba architecture. AiM employs Mamba, a novel state-space model characterized by its -
ExpertsInt8 Quantization Enables Jamba 1.5 Large on Single Node
By
–
To make sure Jamba 1.5 Large fits a single node, we developed ExpertsInt8, a quantization technique for MoE models. We are now able to fit the model with its entire 256K context window on a single node, without losing quality or having a complicated quantization process. [6/6]
-
Jamba 1.5 Models: Hybrid SSM-Transformer Architecture Innovation
By
–
The Jamba 1.5 models are based on our novel hybrid SSM-Transformer architecture, which combines the quality, speed and efficiency of both. Jamba 1.5 Mini has 12B active/52B total parameters, while Large is 94B active/398B total – the largest Mamba model ever made. [2/6]
-
Conda Alternatives for Managing PyTorch and DeepSpeed Dependencies
By
–
please suggest conda alternatives. needs to manage torch, deepspeed, etc without fuss.
-
Jamba 1.5 Mini and Large Models Released on Hugging Face
By
–
Jamba 1.5 Mini and Large on Hugging Face: https://
huggingface.co/collections/ai
21labs/jamba-15-66c44befa474a917fcf55251
…