no it’s very small since we’re looping over each weight matrix individually the only way memory usage would be non-negligible would be if u used a large number of devices — say 256 GPUs in that case don’t use my code 🙂
@jxmnop
-
PyTorch DDP Gradient Syncing Bug in Distributed Training
By
–
so last week I was posting about a bug i had with gradient syncing in torch DistributedDataParallel (DDP), aka the most standard way to do multi-GPU training for neural networks all of my misery originated with a single design decision from the torch team: DDP shares
-
C Language Overhead: Assembly Alternative Discussion
By
–
not C. too much overhead. try assembly instead
-
Pandas DataFrame Cloud Sharing Syntax Innovation
By
–
i want this syntax: > df = pd.DataFrame(…)
> http://
df.to_cloud() https://
website-with-your-df.com/a0283u0j -
Cloud Data Sharing Syntax for Pandas DataFrames
By
–
i want this syntax: > df = pd.DataFrame(…)
> http://
df.to_cloud() https://
website-with-your-df.com/a0283u0j -
Google Docs for Pandas DataFrames: Cloud Collaboration Tool
By
–
who is building the google docs for pandas dataframes? i would use that often. i want to send df links, save them to my notes, plot graphs in the cloud, etc.
-
Intense Research Day Yields One Very Important Discovery
By
–
today was an intense day of research. running commands, writing code, checking numbers, being confused, looking at notes, running more commands, taking a nap, literally going then at the end of the day i figured out one (1) Very Important Thing. so today counts as a good day
-
PyTorch Compile Tutorial: Optimizing Machine Learning Code
By
–
https://
pytorch.org/tutorials/inte
rmediate/torch_compile_tutorial.html
… -
PyTorch Optimization: torch.compile vs Vanilla Performance
By
–
this isn’t comparing to torch.compile, which is the torch-native optimized version, it’s comparing to vanilla pytorch which is notoriously bloated and slow 🙂