HeadInfer: Unlocking Long-Context LLM Inference on Consumer GPUs (Million-level Tokens)
*long-context inputs require large GPU memory.
*A standard LLM like Llama-3–8B requires 207GB of GPU memory for 1 million tokens — far beyond the capabilities of consumer GPUs like the RTX
COMPUTING
-
HeadInfer: Long-Context LLM Inference on Consumer GPUs
By
–
-
ChatGPT Code Interpreter now works with GPT-4.5 and o3-mini
By
–
Just noticed ChatGPT Code Interpreter (the OG, with file downloads; not the in-browser kind) not only works in 4.5, it works in o3-mini too. See example using both. Was this announced? C.I. used to be stuck in 4o which was often too dumb or too slow; now there’s a fix for both
-
QwQ-32B Matches DeepSeek R1 at 20X Smaller Scale
By
–
DeepSeek just got DeepSeek'd. QwQ-32B is on par with DeepSeek R1 671B but ~20X smaller. AGI will run on your computer. Then it'll run on your phone.
-
Cerebras Appoints Hagay Lupesko as SVP AI Cloud Inference
By
–
Please join us in welcoming three exceptional leaders to Cerebras! – Hagay Lupesko, Senior Vice President of AI Cloud and Inference: With deep expertise in software engineering and machine learning, Hagay will accelerate the development of our AI Inference and Cloud
-
Rollout Complete: Major Technology Deployment Ahead of Schedule
By
–
Rollout complete (Faster than expected)
-
Apple Mac Studio 512GB: The Ultimate AI Computing Machine
By
–
Apple's 512GB Mac Studio is no joke. Forget that most PCs max out at 128GB. That's not even VRAM. At 512GB of unified memory you're competing with multiple 5-figure $ GPUs. This can run DeepSeek R1 and basically anything else. There is no better computer for AI.
-
New MacBook Air Sky Blue Color Upgrade Consideration
By
–
Bought a new MacBook Air 4 months. Seriously considering trading in for Sky Blue!
-
SambaNova Cloud expands with SoftBank, Llama and Qwen access
By
–
Making waves We're expanding our SambaNova Cloud deployment with @SoftBank
! Devs will get access to … Swallow, Japan's premier open-source model @AIatMeta
's Llama @Alibaba_Qwen … all through SambaNova Cloud Read the press release #AI