Prompt caching for production LLM inference

2026-09-25 · ai llm performance llm-ops

Prompt caching is a provider- or runtime-level optimization that reuses the model's key-value (KV) state for an unchanged prompt prefix. When a new request starts with the same prefix as a cached request, the system can skip recomputing that prefix and charge discounted cache-read rates instead of full input rates. For production LLM systems with stable system prompts, tool definitions, long reference documents, or repeated agent history, prompt caching can reduce latency and token spend without changing model outputs for the cached prefix.

It is different from application-level response caching. Response caching stores a previous answer and may return it for an identical or semantically similar query. Prompt or prefix caching still runs the model on the uncached suffix; it only avoids reprocessing the cached prefix.

What gets cached

Prompt caching works on prefixes, not arbitrary substrings. The cached unit is the model's internal state for tokens the model has already processed in order. If the prefix changes, the cache entry no longer matches from the point of change onward.

Provider implementations expose this in slightly different ways:

Anthropic documents prompt caching for the Messages API and says cache prefixes follow the request order tools, system, then messages, up to the cache breakpoint. Most request blocks can participate, including tools, system messages, text, images and documents in user turns, and tool use/results. Thinking blocks cannot be marked directly with cache_control, but cached thinking behavior is documented separately for multi-turn tool flows.
OpenAI says prompt caching reuses work when requests share the same prompt prefix and that OpenAI caches the model's full rendered context, including hidden system content, developer messages, tool definitions, and conversation history. Cache reuse requires the entire rendered prefix to match.
Google Gemini states that implicit caching is enabled by default for Gemini 2.5 and newer models and that cost savings are passed on when a request hits caches. Google's current Gemini API caching page describes implicit caching and minimum input token limits per model; it does not describe the same explicit breakpoint API shape as Anthropic or OpenAI.
vLLM implements Automatic Prefix Caching (APC) for self-hosted inference by caching KV cache from existing queries and reusing it when a new query shares the same prefix.

The practical design rule is simple: put stable content first, and append changing content after it.

How providers control cache behavior

The control surface varies by platform, which matters when you build portable clients or compare bills.

Anthropic supports two approaches. Automatic caching adds a single top-level cache_control field and moves the breakpoint to the last cacheable block as conversations grow. Explicit cache breakpoints place cache_control on individual content blocks for fine-grained control. Anthropic documents up to four cache breakpoints, a 5-minute default cache lifetime refreshed on reuse, and an optional 1-hour cache duration at higher write cost. The default minimum cacheable prompt length depends on model and platform; Anthropic's Claude API documentation lists minima such as 512, 1,024, 2,048, or 4,096 tokens depending on the model family.

OpenAI documents prompt caching as enabled by default for supported models. For GPT-5.6 and later in the Responses API, OpenAI describes both implicit and explicit caching. Implicit mode places breakpoints automatically; explicit-only mode uses developer-selected breakpoints and can avoid cache-write charges for changing suffix content after the last selected breakpoint. OpenAI documents a 1,024-token minimum cacheable visible input length for GPT-5.6 and later, with model-dependent minimums for earlier models. Cache writes in the Responses API can use a default 30-minute minimum lifetime at 1.25x the uncached input-token rate, or a 12-hour minimum lifetime at 2.5x. OpenAI states cache-read rates are 0.1x uncached input on most of these models and 0.05x on GPT-6.1 Sol and GPT-6.1 Terra.

Google's Gemini API caching page says implicit caching requires a minimum input token count: 4,096 tokens for Gemini 4 Argon, Gemini 3.8 Flash, Gemini 3.6 Flash, and Gemini 3.1 Pro Preview, and 2,048 tokens for Gemini 2.5 Flash and Gemini 2.5 Pro. It also notes that Gemini 4 agent-style multi-turn flows cache the full previous request only when the new request adds at least 2,050 tokens; model output, thought, and function results count toward that total.

Pricing and break-even math

Prompt caching is not automatically cheaper. A cache write usually costs more than uncached input, while later cache reads cost much less. The workload needs enough reuse before the cache entry expires.

Anthropic's pricing table (fetched 2026-11-07) shows model-specific rates. The general multipliers are 1.25x base input for 5-minute cache writes, 2x for 1-hour cache writes, and 0.1x for cache reads, with per-model exceptions noted in the table (for example, lower cache-hit multipliers on some newer models). Because pricing is model- and platform-specific, production cost models should read the provider's current pricing page rather than hard-coding one multiplier.

OpenAI gives a concrete example for GPT-5.6 and later at the 0.1x read rate with 30-minute writes: writing a prefix once and fully reusing it once costs 1.35x its ordinary input cost, compared with 2x without caching. Across ten requests with one write and nine full reads, the cost is 2.15x versus 10x without caching. That arithmetic is useful, but it assumes the prefix is actually reused and that the workload reaches a machine with a matching cache entry.

Production trade-offs

Prefix stability beats clever prompts. A timestamp, request ID, or per-user preamble inserted near the start of an otherwise stable system prompt destroys prefix reuse. Both Anthropic and OpenAI emphasize that changes before a breakpoint invalidate reuse after that change. Keep volatile fields at the end of the cached prefix or outside it.

Routing affects hit rate. OpenAI notes that cached states live on individual machines and that cache reuse requires reaching a machine holding a matching unexpired entry. On earlier models, OpenAI recommends stable prompt_cache_key values to help route related requests. For GPT-5.6 and later, OpenAI says routing is handled automatically but keys can still separate cache accounting. Production systems should monitor hit rate by tenant, model, endpoint, and prompt shape rather than assuming cache availability.

Minimum lengths create cliffs. Prompts just under a provider's minimum cacheable length produce no cache benefit. Anthropic returns no caching error in that case; you must inspect usage fields such as cache_creation_input_tokens and cache_read_input_tokens. OpenAI and Google publish minimums that differ by model and, for older OpenAI models, by request settings. A small, stable expansion of the prefix can be worthwhile if it crosses the threshold and the prefix is reused often.

Cache invalidation is part correctness, part economics. Anthropic's documentation lists invalidation cases such as tool definition changes, web-search or citation toggles, speed or tool-choice changes, images, and some thinking/effort interactions. OpenAI similarly warns that tools, reasoning settings, text format, verbosity, and compaction can affect the prefix. If your agent framework mutates tool schemas or serializes context differently between turns, cache hit rates can collapse without an obvious application bug.

Self-hosted prefix caching has different constraints. vLLM's Automatic Prefix Caching reuses KV cache across queries with shared prefixes and is enabled with enable_prefix_caching=True. vLLM also documents hybrid Mamba behavior, where prefix-cache hits can resume only at block boundaries unless additional checkpointing is configured, and it has model- and backend-specific limits such as --prefix-match-unit constraints. These details show that "prefix cache hit" is not a generic feature; it depends on the serving runtime, model architecture, and memory layout.

Conclusion

Prompt caching is most effective when the application is designed around stable prefixes: fixed instructions, tool schemas, policy text, and durable context come first; user-specific and time-varying data come last. Before relying on it in production, verify the provider's minimum length, TTL or retention behavior, write/read multipliers, invalidation rules, and usage metrics for the exact model you run. Measure cache-hit rate and end-to-end latency, not just discounted token counts. A high theoretical discount is only real when requests share enough prefix, survive long enough, and reach compatible cache state.

Sources

← back to posts