DDP or FSDP2, depending on the type of model. If it can saturate compute with DDP, then that's simpler and already good. If not, FSDP2.
DDP versus FSDP2 model training strategy selection
By
–
By
–
DDP or FSDP2, depending on the type of model. If it can saturate compute with DDP, then that's simpler and already good. If not, FSDP2.