ok so, it was a gradient bug. let me explain with the huggingface trainer, when training on multiple GPUs, sometimes the model is stored as an nn.Module, but sometimes it’s wrapped in this DistributedDataParallel thing I was inadvertently forwarding on the nn.Module in

→ View original post on X — @jxmnop