re: Infra One of the reasons I focus on Inference Engines/Software Stacks 2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving to vLLM w/ TP=2 RTX PRO 6000: ~32 tok/s → ~110 tok/s moving to Sglang So yeah Edge: llama.cpp > Ollama CUDA / 2+ GPUs: ExLlamaV3/vLLM/Sglang > llama.cpp
Inference Engine Performance: vLLM and Sglang GPU Optimization Benchmarks
By
–
