A document assistant can read the same policy manual all morning and still pay to process it again on every request. The problem might not be the model or the cache setting. It might be a timestamp placed before the manual.
That small detail captures the operational reality of prompt caching: reuse depends on the requests you actually send, not the repetition a human sees in the workflow. Two prompts can contain the same document and still fail to share the prefix the inference system needs.
For engineering teams, the useful question is therefore not “Does our provider support caching?” It is “Have we designed a workload that can benefit from it?”
Cache the computation, not the answer
Prompt caching reuses work already performed on a shared prompt prefix. In a typical transformer serving system, that means reusing key-value state rather than recomputing the same input. The model still generates a response to the current request.
This is different from returning a saved answer to a similar question. It is also different from replaying a previously successful agent trajectory. Those shortcuts reuse outputs or decisions. Prefix caching reuses inference state for matching input.
The distinction matters because it sets expectations. A cache hit does not mean the current answer is already known, the source material is current, or the model's next tool call is authorized. It means some input processing can be skipped.
vLLM's automatic prefix caching documentation makes the boundary explicit: caching reduces prefill work, not the decoding work needed to generate new tokens.
Put stable material before moving parts
Imagine an internal assistant that receives a long operating manual, a fixed set of tool definitions, a current timestamp, and a new employee question. This is a hypothetical workload, but the failure mode is concrete: putting the changing timestamp ahead of the manual makes the prefix diverge before the largest reusable section.
A better starting layout places genuinely stable instructions and shared context first, then request-specific material. Keep tool definitions in a consistent order. Avoid inserting request identifiers into otherwise reusable instruction blocks. Put volatile context where it does not unnecessarily disrupt shared prefixes.
These are layout recommendations, not a universal provider specification. The actual prompt serialization and cache rules still matter.
Anthropic's prompt caching guide describes a hierarchy of tools, system content, and messages. It also explains an easily missed detail about explicit breakpoints: cache entries are written at the marked boundary. The lookup can find earlier entries that previous requests wrote, but it does not discover and cache an arbitrary stable region hidden behind a varying block.
For that API, marking the timestamp block is not equivalent to marking the last stable block before it. An application can repeatedly write new entries while getting little useful reuse.
Treat prompt construction as versioned application behavior. A harmless-looking template change can affect both what the model sees and what the serving layer can reuse. Test both consequences.
Repetition needs a useful time window
A shared prefix is only useful when a later request can reuse it. A manual queried repeatedly within a short session is a different workload from one queried once each morning.
Hosted services may expose explicit cache lifetimes and separate write and read charges. Self-hosted engines manage finite cache capacity and eviction. Neither arrangement justifies assuming that yesterday's warm state will still exist today.
Anthropic documents model-dependent minimum cacheable lengths, lifetime options, and separate usage fields for cache creation and cache reads. Those details belong in an implementation checklist. They should not be flattened into a blanket claim that every repeated prompt becomes cheaper.
Measure the actual reuse pattern before extending retention or adding warmup traffic. Prewarming might help an interactive workload, but it also creates work. If the expected follow-up never arrives, that work has not improved a customer outcome.
Start with natural repetition. Only then consider whether deliberate warming solves a measured cold-start problem.
A fast first token is not a fast workflow
A long document followed by a short answer is a promising candidate for prefix reuse. A short prompt followed by a long report may be dominated by output generation instead. The same cache feature can have very different value across these two workloads.
There is another boundary beyond the model. Search, document loading, tool execution, queueing, and validation all contribute to end-to-end latency. Faster prefill does not make a slow external API faster.
Separate time to first token from time to completed, usable output. Then break down the workflow far enough to identify where a cache hit helps. A latency chart that combines long and short tasks can hide the result you actually need.
Caching is an optimization of a particular phase, not a promise about the entire application.
Share within the right trust boundary
Maximum reuse is not the only goal. In a multitenant application, matching text does not automatically mean two requests should share every optimization boundary.
vLLM's prefix caching design describes optional per-request cache salts that restrict reuse to requests sharing the same salt. Its cache identity also accounts for factors beyond plain text, including multimodal inputs and adapter identifiers.
That is a useful reminder to review the deployed engine's behavior rather than invent a cache key from a visible prompt string. A cache salt is not a substitute for authenticating users or checking document permissions. Choose sharing boundaries deliberately, and enforce application authorization independently.
Freshness deserves the same discipline. Never keep an obsolete policy in the prompt merely to preserve a hit rate. When authoritative content changes, the application should send the correct version and accept the resulting cold work.
Run one controlled workload experiment
Pick one repeatable, read-only workflow before redesigning the whole platform. Use sanitized test inputs and the same task set to compare the existing prompt layout with a stable-prefix layout. Keep model settings and expected answers fixed, and record which requests are cold or warm.
A useful measurement sheet includes:
- Cache-read tokens and cache-write tokens, not just requests labeled as hits.
- Reused input tokens as a share of eligible input tokens, with the denominator stated.
- Time to first token and end-to-end completion latency, including tail behavior.
- Actual billed cost for the workload, including writes and any warmup requests.
- Answer quality, source freshness, and permission checks on the same examples.
Include sparse traffic, a changed document, a changed tool definition, and a cold restart where applicable. Those are ordinary operating conditions, not unfair benchmark cases.
The decision should be boring: keep the change if it improves the target workflow without weakening correctness or isolation. If most time is spent generating output, or requests rarely share a live prefix, another optimization may matter more.
Prompt caching works best when it removes genuinely repeated computation. The engineering job is to make that repetition visible to the serving system, measure the benefit honestly, and leave everything that still needs fresh judgment intact.
Is your agent ready for real work?
The Agent Readiness Kit turns ideas like these into templates you fill in this week: a permission matrix, approval policy, eval suite, failure drills, incident runbook and a 60-point readiness checklist.
Get the Readiness Kit · $19 →