It is harder than you think! It is a fun exercise to consider optimizing deep nets for high cycle times across a datacenter, but it is not at all straightforward, and the “cost of computation” goes way up versus maximizing the work that a single system can do.

→ View original post on X — @id_aa_carmack,