because in distributed mode, each GPU runs its own process. so this code will run simultaneously multiple times, once per GPU. so in this setting, each GPU writes a file, waits for everyone, then aggregates all the files, waits again, then deletes its own file 🙂

→ View original post on X — @jxmnop