That's actually a good point. But it would be quite trivial to restart the cosine schedule after each ~epoch or checkpoint

→ View original post on X — @rasbt