AI Inference Serving: Optimizing Throughput and Latency in Production Systems
Serving large language models in production is mostly a scheduling and memory problem. The model weights are fixed after deployment, but each request creates a growing key-value (KV) cache and arrives at unpredictable times with different prompt and output lengths. A serving stack has to keep expensive accelerators busy without letting time-to-first-token (TTFT), time between generated outputs, or end-to-end latency drift outside service objectives.
The useful metrics make the trade-off explicit. vLLM's metrics documentation defines TTFT as the time from request submission until the first output token, inter-token latency as the gap between consecutive streamed outputs, and per-request time per output token (TPOT) as (end-to-end latency - TTFT) / (number of output tokens - 1). End-to-end latency is roughly TTFT plus decode time; throughput is usually reported as tokens or requests per second across the server. Optimizing one metric in isolation can damage another: larger batches often improve GPU utilization and token throughput, but they can increase queueing delay and tail latency.
Batching turns uneven requests into usable work
Static batching waits for a fixed group of requests, runs them together, and returns results when the batch completes. It is simple, but it wastes capacity when requests finish at different times or when prompt lengths vary. The batch is only as fast as its slowest member, and empty slots cannot be refilled mid-flight.
Continuous or iteration-level batching changes the unit of scheduling. Instead of holding a batch together until every request finishes, the scheduler rebuilds the running batch at decode steps: completed requests leave, and new or waiting requests can join. vLLM documents continuous batching as part of its serving model, and Hugging Face Text Generation Inference (TGI) lists continuous batching of incoming requests as a core feature for increased total throughput. The gain comes from keeping decode steps populated, not from making any single request magically faster.
NVIDIA Triton Inference Server exposes a related but more general mechanism through dynamic batching. Triton's batcher documentation describes dynamic batching as combining inference requests so batches are created dynamically, typically increasing throughput for stateless models. Operators can tune preferred batch sizes, maximum queue delay, priorities, and queue policy. Triton also documents sequence batching for stateful models and iterative sequences for workloads that process one request across multiple scheduling iterations, which is relevant to inflight batching patterns used for generative models.
KV-cache management sets the real batch limit
Batching only works if memory can hold the active requests. During autoregressive decoding, each token produces keys and values that must remain available for later attention steps. The KV cache grows with sequence length and changes over the life of a request, so naive contiguous allocation can leave memory fragmented or reserved but unused.
PagedAttention, introduced by Kwon et al. in Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023), applies virtual-memory-style paging to KV-cache storage. The paper's abstract describes PagedAttention and vLLM as reducing KV-cache waste from fragmentation and redundant duplication, enabling near-zero waste in KV-cache memory and flexible sharing within and across requests. In the authors' evaluation, vLLM improved throughput of popular LLMs by 2–4× at the same latency level compared with FasterTransformer and Orca, with larger gains for longer sequences, larger models, and more complex decoding algorithms. Those numbers are paper results, not universal guarantees; production outcomes depend on model, hardware, prompt shape, and workload mix.
Prefix or prompt caching interacts with serving efficiency because a reused prefix avoids recomputing part of the prefill and can reduce memory or compute pressure for repeated context. It is most effective when applications place stable instructions, tools, and documents before request-specific data. Prefix reuse also changes scheduler behavior: cache hits can reduce TTFT, while cache misses, eviction, or routing to machines without the right cache state can make latency less predictable. Systems should measure both cache behavior and request latency rather than assuming a discount translates directly to better service.
Serving stacks expose different control points
Production teams usually choose between an LLM-focused serving engine and a broader inference platform. vLLM is centered on LLM throughput features such as continuous batching and PagedAttention, and it exposes detailed Prometheus metrics including queue, prefill, decode, TTFT, inter-token latency, TPOT, and KV-cache usage. That observability is useful when tuning SLOs because it separates scheduler delay from model execution.
Triton Inference Server is a general model-serving platform that can host models from multiple frameworks and backends. Its dynamic batcher is useful when you need explicit control over batch formation, queueing, priorities, and model instances across heterogeneous deployments. For LLM workloads behind Triton, the effective behavior depends on the backend and model type: stateless batching, sequence batching, and iterative-sequence scheduling are different tools.
TGI historically provided an inference server for popular open-source models with continuous batching, token streaming, tensor parallelism, and OpenTelemetry/Prometheus support. The Hugging Face TGI documentation now states that text-generation-inference is in maintenance mode and recommends downstream engines such as vLLM and SGLang for ongoing optimized inference work. That status matters when selecting a long-term serving base, but existing deployments should still be evaluated against their measured latency, throughput, and operational requirements.
Capacity planning is an SLO problem
Autoscaling an inference service is not the same as autoscaling a stateless HTTP API. A replica can appear healthy while its scheduler queue grows, KV-cache occupancy approaches saturation, or long prompts cause TTFT spikes. Useful capacity signals include queue depth, running/waiting request counts, KV-cache usage, admission rejections, TTFT, inter-token latency, TPOT, and end-to-end latency distributions. vLLM's metrics design explicitly treats request-level latency metrics as service objectives and uses server-level metrics to explain them.
Load tests should reflect production shape: prompt length distribution, output length distribution, concurrency, arrival bursts, and prefix reuse. A workload with shared system prompts may benefit strongly from prefix caching; a workload with unique long prompts may be limited by prefill cost and memory. Before adding replicas, check whether the bottleneck is compute, memory, scheduler policy, model parallelism, or request routing. Scaling replicas helps only when new capacity can accept work without reproducing the same queueing or cache-miss behavior.
Conclusion
Good inference serving comes from matching scheduler policy, memory management, and capacity signals to the workload's latency budget. Continuous or dynamic batching improves utilization when requests can be combined safely. KV-cache techniques such as PagedAttention and prefix reuse raise the effective batch limit, but they also add statefulness and invalidate simple throughput assumptions. Measure TTFT, decode latency, throughput, and queue behavior under realistic traffic before treating any benchmark or pricing claim as a production guarantee.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention (Woosuk Kwon et al., SOSP 2023)
- vLLM Metrics design documentation
- NVIDIA Triton Inference Server: Batcher
- Hugging Face Text Generation Inference documentation
- Prompt caching for production LLM inference (related post on this site)