To run MiniMax M5 on a Mini M5 Max locally
@rasbt
-
RLVR Sequel with GRPO and Distillation Techniques
By
–
Thanks! You might also like the sequel on RLVR with GRPO and distillation: https://
github.com/rasbt/reasonin
g-from-scratch/tree/main/ch06
… -
DeepSeek V3.2 Uses Mixture of Experts Together
By
–
yeah they are used together like in DeepSeek V3.2
-
MLA Uses DSA Underneath According to DeepSeek V3.2
By
–
I'd say it the other way around: MLA uses DSA underneath, but that's right! E.g., from the DeepSeek V3.2 paper:
-
DSA: Learning-Based Token Selection Beyond Fixed Window Attention
By
–
Also, DSA is basically a smarter version of sliding window attention where you "learn" which past tokens to select versus forcing it to be in a specific window
-
DeepSeek V3.2 Success: Why Not Apply This Approach
By
–
I mean why not? It worked pretty well in DeepSeek V3.2
-
Question about sliding window attention implementation
By
–
Hmm I don’t think they use sliding window attn ? Where did you see that?
-
Expert Size and Number Match DeepSeek V3 Specifications
By
–
Probably also should have added that
– the expert size (2048) and number (256) is now exactly the same as in DeepSeek V3 / V3.2. -
GLM-5 Architecture: Weights Released, Key Features Unveiled
By
–
The weights are out! Here's the GLM-5 architecture comparison. GLM-5 is: – bigger than its predecessor (mainly more experts) but has rel. similar active parameter counts – uses multi-head latent attention – uses DeepSeek Sparse Attention
