There was a bug with gradient norm clipping, it was incorrectly synchronized across GPUs when using ZeRO-1, which may have contributed to the loss spike I saw earlier. It's fixed now on master. More generally as of yesterday though we've re-established full and exact equality to

→ View original post on X — @karpathy