But those were also much much bigger runs, so it's a lot more impressive. This was on a single node so you don't need to deal with any cross-node interconnect. It starts to get a lot more fun when you have to keep track of O(10,000) GPUs all at once. For a very specific

→ View original post on X — @karpathy,