Caching
Agents repeat themselves constantly: the same document read four times in a run, the same question asked by ten people in an hour. Caching is the cheapest cost reduction available, and the only hard part is knowing what is safe to serve stale.
How it works
- Cache tool results within a run by default. The same read twice in one run is almost always a bug anyway.
- Cache across runs only where the underlying data has a known change rate.
- Use provider prompt caching for the stable prefix: system prompt, tool definitions, and long context.
- Key on the exact input, and include the version of the prompt and the tool definitions in the key.
- Never cache anything user-specific in a shared cache. This is where caching becomes a data breach.
Optional: load a real model
See it work
When it pays, and when it does not
| Use it when | Skip it when |
|---|---|
| The same inputs recur | Every request is genuinely unique |
| Data changes on a known schedule | Data changes unpredictably and staleness is dangerous |
| Prompts have a long stable prefix | Prompts are short, where prompt caching saves little |
| You can key precisely | Keys would have to include everything, which means there are no repeats |
How it fails
Cross-user leakage
The worst caching bug there is, and it is always a key design problem.
Stale after a deploy
A prompt change with an unchanged cache key serves yesterday's behaviour. Version the key.
Caching the wrong layer
Caching a whole agent run hides the fact that one tool result is stale inside it.
What it costs
| Savings | Often 40 to 70% of model spend in a repeat-heavy workload. The largest lever most teams have. |
|---|---|
| Complexity | Moderate, and the failure modes are subtle. |
| Risk | Concentrated entirely in the key design. Get that right and the rest is easy. |
Version your cache keys with your prompts. The strangest agent bugs come from a cache serving the behaviour of a prompt you deleted last week.
Related: Cost control · Budgets and limits · Cost reporting · all patterns · agent jobs
Related: Cost control · Budgets and limits · Cost reporting · all patterns · agent jobs
Free from AI School - no signup, everything runs in your browser.