You can always make a batch large enough that the GPU stays busier than the CPU, but if you want to do hundreds or thousands of steps a second, CUDA graphs are really helpful. You should never be rebuilding graphs — the trick is to make sure the one graph you build at startup
@id_aa_carmack
-
CUDA Graphs as Improved Display Lists for GPU Computing
By
–
I feel CUDA graphs are kind of “display lists done right” — if display lists had locked in all the state switches at capture time, and the only thing you could do was update the contents of the referenced buffers, they would have been great. Of course, display lists were there
-
Expressing Hyperparameters Using Negative Integral Log2 Scale
By
–
Sometimes I see papers with hyperparameter sweeps over 0.001, 0.003, 0.006, 0.01, etc. Many hyperparameters are better expressed in negative integral log2. Small values like learning rates directly, and values close to 1 like EMA factors and TD lambda / gamma with 1-2**val. It
-
Continuous Learning Improves Model Credibility Through Linear Source Reading
By
–
If we sort out continuous learning such that large models don’t just get a batch of snippet stew out of a blender, and read sources linearly, it may help with credibility assessment.
-
Blindsight by Peter Watts: A Grand and Challenging Science Fiction Novel
By
–
Blindsight by Peter Watts (2006) is my standout fiction book of the year, by far. Grand, challenging, scary, and super autistic.
-
Environmental Interaction Essential for Learning Complex Tasks
By
–
Which needed huge amounts of interaction with the environment — they weren’t just given the rules and told to think hard about it.
-
AI Value Creation Limited by Rhetoric Without Experimental Science
By
–
I expect AI to produce a lot of value while trying to make a coherent body of knowledge out of its training data, but on its own, AI is limited to a sort of ancient Greek philosophical process, where the method is rhetoric, rather than the experimental process of science. This
-
NumPy’s Simple File I/O API and PyTorch Tensor Limitations
By
–
Numpy’s fromfile()/tofile() make me smile — no typing or byte order metadata, they just write/read the raw bytes. A joyful API! I wish PyTorch had them natively on tensors, so I wasn’t doing so many tensor.cpu().numpy() operations.
-
Weight decay ineffective for value-based reinforcement learning
By
–
No, I haven’t seen any reliable wins from weight decay for value based RL, and I haven’t looked several times.
-
Hyperparameter Changes for Muon Optimization and Optimizer Comparison
By
–
What hyperparameters changed when optimizing for muon? @clashluke has tried several new optimizers on our code and not reliably beaten Adamw.