None of these: just PyTorch with distributed training and tensor sharding. Optionally with CPU offloading for really big LLMs. I use Fabric as a convenient wrapper here.
PyTorch Distributed Training and Tensor Sharding with Fabric
By
–
By
–
None of these: just PyTorch with distributed training and tensor sharding. Optionally with CPU offloading for really big LLMs. I use Fabric as a convenient wrapper here.