Tensor Parallelism is a bandwidth multiplier btw That’s why stacking Mac Studios / DGX Sparks / GPUs increases tokens/second (The rate at which the bandwidth is multiplied is what differentiates Unified Memory from VRAM as well)

→ View original post on X — @theahmadosman,