While async Tensor Parallelism is common among the elite private codebases, the PyTorch team put together a public, accessible and readable one. pretty cool work from @foofoobuggy @cHHillee @lessw2020 @lc_whr tianyu and @wanchao_ Read more here: https://
discuss.pytorch.org/t/distributed-
w-torchtitan-introducing-async-tensor-parallelism-in-pytorch/209487
…

→ View original post on X — @soumithchintala,