Nice! thanks for the new release (and also for constantly improving mps support)!
@rasbt
-
Residual Attention and DeepSeek V4 Developments
By
–
Yes, next to residual attention as hot new candidate. And, I am also of course curious to see what DeepSeek V4 is up to haha
-
Cross Attention in Separate Articles on Multi-Modal Models
By
–
Yeah, I have cross attention in my separate articles (the multi-modal and understanding attention from-scratch) ones. This article was focused on text LLMs to keep the scope reasonable.
-
XSA Attention Mechanism: Future Research Candidate for Follow-up Study
By
–
Nice one! XSA from https://
arxiv.org/abs/2603.09078? To keep the scope reasonable, I only focused on those that already made it into the flagship architectures. But maybe that's an interesting candidate for a follow-up on attention research candidates. -
Olmo Hybrid Architecture and Jamba as Nemotron 3 Predecessor
By
–
I think Olmo hybrid also uses Gated DeltaNet though right? Jamba is a good point, I think that’s kind of like a predecessor of Nemotron 3 (with mamba 1 instead of 2) if I recall correctly. It’s been a while
-
Understanding Your Tools and Data Matters More Than Memorization
By
–
Yes, no need to memorize things but it's useful to understand what you (or your agent) is working with
-
Visual Guide to Modern LLM Attention Variants
By
–
A visual guide to modern LLM attention variants, all in one place: magazine.sebastianraschka.co…
-
Added XLSTMb Architecture to LLM Gallery
By
–
Added https://
sebastianraschka.com/llm-architectu
re-gallery/#card-xlstm-7b
…
Thanks again! -
Open-Source Models: Attribution and Community Adoption
By
–
Interesting! Either way let’s not discourage adopting and tuning open-weight & open-source models; that’s what they are for.
But yeah, open-source licenses usually have an attribution clause…