5/5 Because this approach is model-agnostic, it applies to any architecture. Even on transformers (like Qwen2.5-7B by @alibaba_cloud
, this method recovers ~90% of the gains of sequence packing, without relying on specific attention implementations. Full breakdown +

→ View original post on X — @ai21labs,