The claims (vague as they are) are implausible — bandwidth and computation in production models are such that 20x+ improvements and 99% energy savings are just not on the table. There are plenty of improvements available, but none that go that far. Rather than just dueling
@id_aa_carmack
-
CUDA Graphs Eliminate Python and C++ Overhead Benefits
By
–
CUDA graphs get rid of both python and C++ overhead; I’m a fan!
-
PyTorch Built-in Kernels Outperform Custom Function Series
By
–
For writing your own kernels, sure, but a PyTorch built-in is almost always going to be faster than an implementation consisting of a series of simpler PyTorch functions.
-
AI Models Need Help With Unprompted Volunteering Suggestions
By
–
“volunteering suggestions” would be super helpful, and it is one of the things human experts are still superior at, but it would be hard to train models for. We are sort of built around prompting, so unprompted suggestions aren’t really in domain.
-
PyTorch Built-in Functions Outperform Custom Tensor Rearrangement Code
By
–
Always a slightly mixed feeling to write pretty good first-principles code to do some tensor rearrangement, only to find that PyTorch has a built in function that does it faster. I had made a point of at least skimming the docs of every torch and tensor function, but if you
-
Softmax in Base 2: Fixed Point Hardware Optimization
By
–
I wonder if doing softmax in base 2 instead of base e could be a useful optimization on some fixed point hardware. Prepend a 1 to the fractional part and shift left by the integral part as a close-ish approximation of 2^fixed_point.
-
Hardware Overhang: Why Neural Networks Stayed Slow in the 90s
By
–
There were good matmul libraries in the 90s, but so many researchers were implementing their 20 hidden neuron networks in matlab loops. There was a hardware overhang for quite some time.
-
Appreciation for Grok’s Direct Correction Approach
By
–
I appreciate how Grok doesn’t sugar coat corrections.
-
Remote-Operated Robots: A Realistic Alternative to Autonomous Household Humanoids
By
–
Companies selling the dream of autonomous household humanoid robots today would be better off embracing reality and selling “remote operated household help”. Have teams of employees running them 24/7, with the option to reduce their workload as autonomous behaviors become viable.
-
DGX Spark Performance Issues Power Draw and Thermal Concerns
By
–
DGX Spark appears to be maxing out at only 100 watts power draw, less than half of the rated 240 watts, and it only seems to be delivering about half the quoted performance (assuming 1 PF sparse FP4 = 125 TF dense BF16) . It gets quite hot even at this level, and I saw a report