I think so. In their 3.35B Tiny Aya report, they say "We use parallel Transformer blocks, which lead to a signifi-
cant improvement in training efficiency without hurting model quality."
@rasbt
-
Technical training efficiency insights from the Tiny Aya model report
By
–
-
Technical Analysis of Parallel Block Design in LLM Architectures
By
–
It's been *almost* a bit quiet around LLM architecture releases in the past two weeks Interesting tidbit is the parallel block design. Via the Cmd-A the tech report "equivalent performance but significant improvement in throughput compared to the vanilla transformer block."
-
A Visual Tour of Recent LLM Architecture Advances
By
–
New article: a visual tour of recent LLM architecture advances, from Gemma 4 to DeepSeek V4. I focus on long-context efficiency tweaks like KV sharing, per-layer embeddings, layer-wise attention budgets, compressed attention, and mHC. Link: https://
magazine.sebastianraschka.com/p/recent-devel
opments-in-llm-architectures
… -
Comparative analysis of GQA, MLA, and Gated DeltaNet model architectures
By
–
Yes! It would be interesting whether this happens in GQA:SWA models only though or also in MLA or Gated DeltaNet models, for example.
-
Learning LLM Architectures Through Scratch Implementation and Model Evaluation
By
–
A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch. And how I approach new open-weight models, compare them against reference implementations etc:
-
Low-commitment attention modification for model training
By
–
Interesting paper. What I like about this is that it is a relatively low-commitment attention modification. I.e., one can use it during most of training, switch back to vanilla attention near the end, and recover roughly the same modeling performance as if full attention had
-
Technical Requirements for Evaluating AI Model Architecture
By
–
Thanks, but there are no open weights yet, right? Asking because it would be impossible to cover architecture details without open weights and/or a detailed technical report
-
State Space Models in Gallery Updates
By
–
I have models with state space layers in the gallery (e.g., Nemotron 3). Anything since April I missed though?
-
Shared Experts Reduce Redundancy in Mixture-of-Experts Models
By
–
It can learn shared patterns so that the individual experts don’t have to relearn the same info; ie it’s to reduce redundancy among the non-shared experts
-
April 2025 AI Architecture Drops: Six New Models Released
By
–
Here is a 2nd batch of April architecture drops. What a month!
– Ant Ling 2.6 1T
– Minimax M2.7
– Xiaomi MiMo V2.5
– Poolside Laguna XS.2
– Tencent Hy3-preview
– IBM Granite 4.1