Small correction: prefill is compute-bound, decode is the memory-bandwidth-bound phase. But the KV cache point is spot on. It grows linearly with context, and every decode step has to read the whole thing.
MACHINE LEARNING
-
Techniques to Improve AI Sequence Decoding Efficiency
By
–
Yes, and it is linear in sequence length, so the pain shows up fast. This is why GQA, sliding windows, and quantized caches exist. They all attack the same problem from different angles: keep the cache small so decode stays fast.
-
Every ML conference since 2019 features at least three subquadratic attention papers
By
–
every ML conference I've been to since 2019 has had no fewer than three papers proposing new techniques for subquadratic attention
-
New sub-quadratic attention technique makes long-context LLMs 10x cheaper
By
–
"Introducing a breakthrough new technique for sub-quadratic attention, making long-context LLMs 10x cheaper without sacrificing performance" Me:
-
Zyphra small model impressive but reasoning still inaccurate
By
–
I love Zyphra, and it's amazing how much they can do in a small model but the reasoning is not yet ready (extremely long winded but still inaccurate). I am looking forward to the next update
-
Anthropic releases API for Claude’s memory-organizing Dream feature
By
–
Yesterday, Anthropic just released the API for this Dream, which is really fascinating. What it does is let the Claude agent "dream" on its own, organizing the memories it wrote down in past sessions—merging duplicates, updating expired ones, unifying contradictions, and even
-
Subquadratic Attention and Data Quality: Challenges for Large Context AI Models
By
–
people on here are dumb. the latest subquadratic attention trick might produce a model that *processes* 1M tokens (or 12M..) without going insane, but that doesn't make it good the real problem isn't the architecture, it's the data. humans haven't generated many contiguous
-
Epstein suicide note OCR benchmark test results
By
–
Epstein Suicide Note OCR Benchmark test: 1. Gemini 85%
2. GPT-5.5 82%
3. Grok 4.3 80%
4. Opus 4.7 75% -
Learn Pragmatic Machine Learning Deployment with Python
By
–
Two South Korean companies – SK Hynix and Samsung – charting new territory. Unbelievable value being captured in the ‘Intelligence’ supply chain.
