This new optimizer can make training giant LLMs both more stable and more precise, even under noise and extreme scale! Huawei just introduces ROOT, a Robust Orthogonalized Optimizer that tackles two big weaknesses in recent momentum-orthogonalized methods: – Dimensional

→ View original post on X — @jiqizhixin,