one more thing to add.
at this scale, we also have to adjust the actual packet routing algorithms in our switches and NICs, to be able to load-balance well. Did you know switches have to have significant HBM memory as well (not just GPUs) because as the packets queue up, they
HBM Memory in Switches: Critical Infrastructure for AI Scale
By
–