It's been a few months but I didn't see a noticeable speed difference tbh. Otherwise, Ollama has usually a nice set of quantized models and easy to switch.
@rasbt
-
Decode 80 tok/sec at 1k, prefill 3000 tok/sec at 50k contexts
By
–
Yeah, this was at 50k contexts. Decode is about 80 tok/sec at 1k contexts. Prefill is up to 3000 tok/sec at
-
Question about Pi’s selling point over Cline for classic codex user
By
–
Thx. As someone who mostly uses classic codex, what's the like the selling point behind Pi over Cline etc?
-
GLM 5.0 report and IndexShare: only notable aspects
By
–
I was thinking about it, but beyond the original GLM 5.0 technical report, I don't think there's anything interesting to write about (except that it's better; and IndexShare, which is a rel simple concept)
-
Not yet tried Cline & Pi, comfortable with Codex due to muscle memory
By
–
No, not yet. There's also Cline & Pi I still have to try. (I am kind of comfortable with codex because of muscle memory.)
-
Local open-weight LLMs: 30B MoE sweet spot at 40 tok/sec
By
–
Have been taking different local open-weight LLMs for a test drive in different harnesses (Qwen-Code, Codex, Claude Code). 30B Mixture-of-Expert models are kind of a nice sweet spot and can solve challenging problems. And they get roughly 40 tok/sec on a Mac or DGX Spark, which
-
Reflection on model distillation and the performance of GLM 5.2
By
–
Not sure, honestly. If that's the case, it wouldn't surprise me, it's quite common. Claude distills from the internet & potentially others, others distill from Claude,… it's just the natural development cycle. GLM 5.2 is >10 points better
-
Debate on most capable open-weight LLM averaged across benchmarks
By
–
I don’t disagree. Let’s say most capable open-weight LLM when averaged over all major benchmarks (reasoning, coding, logic, tool use, math, knowledge)
-
GLM-5.2: best open-weight model with multi-head latent attention
By
–
Just updated with the recent release of GLM-5.2. The best open-weight model today. Architecture-wise, it is built on the GLM-5 and GLM-5.1 architecture that I covered previously, meaning it reuses the Multi-head Latent Attention mechanisms.
-
Nuance in ‘open-weight model’: OpenRouter vs Ollama
By
–
In general, I agree. But they are related but slightly different things in some contexts. Like when I say "I am running an open-weight model", I could be using OpenRouter or ollama cloud models.