2. Sacrificing throughput for latency
Originally, they focused on TTFT (Time To First Token), but realized that TBT (Time Between Token) hurt them more, especially with Chain-of-Thought queries where users don’t see the intermediate outputs. They found that TTFT and TBT
SYSTEMS
-
Trading Throughput for Latency: LLM Performance Optimization
By
–
-
Kubernetes Scaling Limitations for AI Infrastructure
By
–
I'm worried you can't scale down your Kubernetes to this level
-
DistBelief Legacy: Model and Data Parallelism in Modern Neural Networks
By
–
I'm still a bit surprised that you think the system (DistBelief) that first combined large scale model- and data- parallelism with backprop-based training of very large neural nets is a "dead end", given that nearly every large-scale model that is trained today combines these
-
Selective Model Weight Updates for Production AI Systems
By
–
Definitely a promising direction, but it doesn't look to me like that technique is useful for production scenarios yet Being able to selectively update model weights to tell a model "the new CEO of Fiat is X" would indeed be incredibly useful if it was fully reliable
-
LLMs Structured Output as Postel’s Law Implementation
By
–
LLMs that implement constrained sampling/schema respecting structured output seem the ultimate implementation of Postel's Law: "Be conservative in what you send, be liberal in what you accept." Basically the ultimate interface block between systems.
-
MATLAB and Simulink Historical Documentation Archives Discovery
By
–
Found in the archives…MATLAB and Simulink prints
-
Unreliable Computer Systems: Navigating Modern Computing Challenges
By
–
We somehow managed to invent the world's most unpredictably unreliable computer systems! They're still incredibly useful, but you have to know all sorts of tricks to stay out of the many traps they unwittingly set for you
-
Engineering LLM Systems with External Tools for Reliability
By
–
But in a well-engineered system that provides access to tools, runs searches against external systems, then feeds those results back into the LLM as part of the context supplied alongside the prompt: the reliability shoots up enormously… but there will still be weird edge-cases
-
Advanced AI Systems Struggle with Basic Instruction Following
By
–
It's just so weird and unintuitive having super-advanced computers that can't reliably do the things that we tell them to do
-
Compound AI Systems Outperform Single Models Approach
By
–
Compound AI systems bring models, feedback, external knowledge together; going to be much better than just single models/ single shot approaches!