Partial offloads wouldn’t work because this will create a severe bottleneck That’s why the full model has to be offloaded to the GPU to see actual performance gains
COMPUTING
-
GPU Offloading Offers 1800Gbps vs DGX Spark’s 273Gbps
By
–
Yeah that is definitely faster because you’re fully offloading to the GPU which runs at ~1800Gbps in comparison to DGX Spark’s 273Gbps
-
FeatureBench: Evaluating LLM Coding Agents on Complex Software Features
By
–
How well do LLM coding agents truly perform on complex, end-to-end software feature development? Researchers from the Institute of Automation, Chinese Academy of Sciences and Huawei Technologies Co., Ltd. introduce FeatureBench, a new benchmark using a scalable, test-driven
-
New Website to Benchmark Hardware Tokens per Second Performance
By
–
Update: @AlpinDale and I agreed to collab on a website that does more accurate calculations for your hardware tokens/sec performance Expect more on this after I am done with GTC this week
-
Use MoE Models on Unified Memory Hardware Like DGX Spark
By
–
As I have mentioned before, stop trying to get Dense models running on the DGX Spark/Mac Studios Unified Memory is best fit for MoEs because you only make each token go through a small subset of the numbers of parameters in the model Optimize for your hardware
-
27B Parameters Per Token Explains Slow LLM Inference Speed
By
–
Ultimately you’re still going through the 27B parameters per token and that’s what takes so long
-
DGX Spark Qwen3 27B Inference Speed Discrepancy Questioned
By
–
Also, I don’t know how OP is getting Qwen3.5 27B @ 30 tps on the DGX Spark Doesn’t make sense for a Dense model on DGX Spark’s Unified Memory (273 Gbps) (My personal experiments showed it’s 4 tokens/sec) Very curious how you got that @TeksEdge
-
Qwen3 27B Performance Claims Disputed on DGX Spark Hardware
By
–
Also, I don’t know how OP is getting Qwen3.5 27B @ 30 tps on DGX Spark That number is impossible for a Dense model on DGX Spark’s Unified Memory (273 Gbps) My personal experiments showed it’s 4 tokens/sec for that model on the Spark
-
Unified Memory Faster for Loading Large MoE Models
By
–
the issue is that unified memory would still be faster for loading MoEs that are larger than the largest single GPU in terms of Memory
-
RTX PRO 6000 GPU Inference Faster Than Unified Memory After Loading
By
–
This will probably be great for Large single GPUs (e.g. RTX PRO 6000) You’re limited to 40Gpbs initially (during model loading) but then once the model is fully loaded on the GPU it should be extremely faster than Unified Memory speeds for inference