ok good point and i AGREE that the details matter but these kids aren't trying to replicate T5 pretraining in 20 hours on a single gpu; they just want to train little neural networks from scratch; and for their use cases adafactor vs adam won't almost ever matter
@jxmnop
-
Neural Networks Learn Regardless of Input Encoding or Hyperparameters
By
–
one point i try to get across in Practical Deep Learning is that neural networks will LEARN no matter what you do (provided you avoid huge errors like setting all input data to zero or something) any input encoding, any hidden size, any reasonable learning rate…. it will learn
-
Python Random Module: Built-in Alternatives and Use Cases
By
–
what's wrong with just using built-in python random.random() ?
-
PyTorch torch compile encountered critical error during execution
By
–
welp. this is what happened when i tried to use torch compile
-
Parallel Batch Processing Optimization in Machine Learning
By
–
what would that even mean? why would you not just combine “parallel” batches into a bigger batch?
-
Weight Decay Impact on Model Training Speed
By
–
what does increasing weight decay do in this case? I’d expect that to slow training down
-
GPU utilization troubleshooting: why only 30% despite full memory
By
–
yet here's proof that my GPU utilization is still trash. if forward or backward is ~always happening according to the profiler, and memory is decently full now (at least over 50%)… …then how is my GPU being used only on average 30% of the time? what am i missing here?! help
-
Autograd Backward Pass Implementation Progressing Successfully
By
–
and here's a slightly less-zoomed backward pass. autograd is running, backward is happing, etc. etc. seems all fine and good to me
-
PyTorch GPU Utilization Issues: Profiling Forward Backward Pass
By
–
alright y'all. still having GPU utilization issues and against my better judgement i broke out the PyTorch profiler. posting a few screenshots. doesn't seem like I/O is an issue (no gaps) how is GPU util so bad (~30%) if either forward or backward is running almost always?