And there you go! Model weights across GPUs have been synchronized. While the total elements transferred is still the same as we had in the “single-GPU-master” approach, this ring approach is much more scalable since it does not put the entire load on one GPU. Check this

→ View original post on X — @akshay_pachaar