There was some comparison of training performance by @StasBekman recently that showed bf16 non-mixed training was quite a bit worse. Which I'd expect, because bf16 has much less precision than even fp16.

→ View original post on X — @jeremyphoward