I think Olmo hybrid also uses Gated DeltaNet though right? Jamba is a good point, I think that’s kind of like a predecessor of Nemotron 3 (with mamba 1 instead of 2) if I recall correctly. It’s been a while
@rasbt
-
Understanding Your Tools and Data Matters More Than Memorization
By
–
Yes, no need to memorize things but it's useful to understand what you (or your agent) is working with
-
Visual Guide to Modern LLM Attention Variants
By
–
A visual guide to modern LLM attention variants, all in one place: magazine.sebastianraschka.co…
-
Added XLSTMb Architecture to LLM Gallery
By
–
Added https://
sebastianraschka.com/llm-architectu
re-gallery/#card-xlstm-7b
…
Thanks again! -
Open-Source Models: Attribution and Community Adoption
By
–
Interesting! Either way let’s not discourage adopting and tuning open-weight & open-source models; that’s what they are for.
But yeah, open-source licenses usually have an attribution clause… -
Redbubble LLM Architecture Gallery Poster Medium Size Available
By
–
The Redbubble poster version just arrived (redbubble.com/i/poster/LLM-A…)! This is the Medium (26.9 x 23.4 in). Font looks good and sharp, but I probably wouldn't go smaller.
-
Mamba-3 Release: Linear Models for Hybrid Architectures
By
–
Oh wow, Mamba-3 is here! For me, the most interesting use case of Mamba and Mamba-likes are the recent transformer attention hybrid architectures (Qwen3.5, Kimi Linear, etc.) Would be interesting to swap Gated DeltaNet with Mamba-3 (which now also has RoPE) in next gen hybrids. Albert Gu (@_albertgu) The newest model in the Mamba series is finally here 🐍 Hybrid models have become increasingly popular, raising the importance of designing the next generation of linear models. We've introduced several SSM-centric ideas to significantly increase Mamba-2's modeling capabilities without compromising on speed. The resulting Mamba-3 model has noticeable performance gains over the most popular previous linear models (such as Mamba-2 and Gated DeltaNet) at all sizes. This is the first Mamba that was student led: all credit to @aakash_lahoti @kevinyli_ @_berlinchen @caitWW9, and of course @tri_dao! — https://nitter.net/_albertgu/status/2033948415139451045#m
-
Framework-specific challenges with VRAM, disk, and quantization options
By
–
Thanks for the feedback. The only problem with VRAM, disk, and quant options is that they are framework specific. Disk is maybe the most obvious or feasible one for a given precision (like bf16)
-
Inference VRAM Complexity Depends on Framework Implementation
By
–
I was thinking about inference VRAM as well, but this one is so tricky because it depends on the implementation/framework one is using