It feels like they're talking about "offline" distillation here, but actually implemented online distillation as described in the paper

→ View original post on X — @maximelabonne,