Caching, Batching & Model Routing

The three architectural levers that cut bills 50-90% without touching a single prompt: cache what repeats, batch what can wait, route to the smallest capable model.

Lever 1: Prompt Caching - Stop Paying Full Price for Repetition

Most production traffic re-sends mostly identical content: the same system prompt, the same tool definitions, the same reference documents - thousands of times an hour. Prompt caching lets the provider reuse the processed prefix, and charges cached input tokens at roughly 10% of the normal price (mechanics and discounts vary by vendor).

What it takes to win at caching:

• Structure for stability. The prefix must be byte-identical call-to-call: no timestamps, no user names, no request IDs in the system prompt. Volatile data goes at the end (the stable-prefix layout from Lesson 4).
• Mind the TTL. Caches expire in minutes. Steady traffic keeps them warm; sporadic traffic re-pays the full price on each cold start.
• Measure the hit rate. Vendors report cached vs. uncached tokens in every response. A cache hit rate under 70% on a high-volume route means structural problems worth fixing.

Lever 2: Batch Processing - a Flat ~50% Discount for Patience

Major providers offer batch APIs: submit jobs asynchronously (results within hours, often minutes) for roughly 50% off both input and output tokens. Anything not user-facing belongs in a batch:

• Nightly classification and tagging runs
• Embeddings refresh and re-indexing
• Evaluation suites and regression tests
• Report generation and content pipelines
• Backfills and data migrations

Lever 3: Model Routing - the Right-Sized Model for Every Call

The most expensive habit in production AI is sending everything to the frontier model. Sort each call by what it actually needs:

Routing strategies, simplest first:

1. Static routing - route by endpoint or feature. No intelligence needed; start here. Most of the win comes from this step.
2. Heuristic routing - rules on input length, keywords, or user tier.
3. Cascade - try the cheap model first; escalate to a bigger one on low confidence or failed validation.
4. Learned routing - a small classifier picks the model per request. Only worth it at serious scale.

Stack the Levers

The levers compound because they apply to different parts of the bill:

Baseline (everything on frontier, no caching, no batching)   $100,000/mo

1. Route 70% of calls to mid-tier (75% cheaper on those)      -$52,500
2. Enable caching on remaining traffic (60% of input,
   90% hit rate, one-tenth price)                             -$12,000
3. Move offline work (30% of what's left) to batch (50% off)   -$5,300

Optimized total                                              ~$30,200/mo
Reduction                                                        ~70%

Numbers vary by workload mix, but 70-90% reductions versus the naïve baseline are routinely achievable with exactly these three moves - no prompt rewrites, no quality loss, no product changes visible to users.