one point i try to get across in Practical Deep Learning is that neural networks will LEARN no matter what you do (provided you avoid huge errors like setting all input data to zero or something) any input encoding, any hidden size, any reasonable learning rate…. it will learn
@jxmnop
-
Python Random Module: Built-in Alternatives and Use Cases
By
–
what's wrong with just using built-in python random.random() ?
-
PyTorch torch compile encountered critical error during execution
By
–
welp. this is what happened when i tried to use torch compile
-
Parallel Batch Processing Optimization in Machine Learning
By
–
what would that even mean? why would you not just combine “parallel” batches into a bigger batch?
-
Weight Decay Impact on Model Training Speed
By
–
what does increasing weight decay do in this case? I’d expect that to slow training down
-
Autograd Backward Pass Implementation Progressing Successfully
By
–
and here's a slightly less-zoomed backward pass. autograd is running, backward is happing, etc. etc. seems all fine and good to me
-
GPU utilization troubleshooting: why only 30% despite full memory
By
–
yet here's proof that my GPU utilization is still trash. if forward or backward is ~always happening according to the profiler, and memory is decently full now (at least over 50%)… …then how is my GPU being used only on average 30% of the time? what am i missing here?! help
-
Inside BERT: Forward Pass Architecture and Operations
By
–
here's a zoomed in forward pass. it's classic BERT stuff. linear layers, matmuls, softmaxes, some other stuff
-
PyTorch GPU Utilization Issues: Profiling Forward Backward Pass
By
–
alright y'all. still having GPU utilization issues and against my better judgement i broke out the PyTorch profiler. posting a few screenshots. doesn't seem like I/O is an issue (no gaps) how is GPU util so bad (~30%) if either forward or backward is running almost always?