Some intuition, some luck. We already knew that neural net training was very tolerant of noise (we were doing asynchronous updates from dozens of replicas to train large models, which added lots of noise to the updates). Noise is ~ the least important bits, so I said "well, I'd

→ View original post on X — @jeffdean