What Is Prompt Caching? The Complete Guide to Anthropic, OpenAI, Gemini & Grok Caching

Prompt caching (also called prefix caching) is a billing discount every major LLM provider now offers: when the beginning of your prompt — the prefix — is identical to one the provider processed recently, they skip re-processing it and bill those tokens at a fraction of list price, typically around 10%. Your model still runs and produces a fresh answer; you just stop paying full price to re-read the same system prompt, tool definitions and conversation history on every call.

That matters because modern LLM traffic is extremely prefix-heavy. An agent with a 6,000-token system prompt and tool schema resends those tokens on every turn. A support bot resends the whole conversation so far. In practice 60–90% of the input tokens you pay for are tokens the provider has already seen. Prompt caching is the single biggest cost lever most teams have — if the cache actually gets hit.

How prefix matching works

The provider hashes your prompt from the start, in blocks. If the first N tokens match a cached entry byte-for-byte, those N tokens are billed at the cached rate and the model's attention computation for them is reused (which is why cache hits also reduce time-to-first-token — the prefill step is skipped). Everything after the first difference is processed and billed normally.

Two properties follow, and they explain nearly every “why is my hit rate zero” mystery:

• Matching is exact and positional. One changed character at position 100 invalidates everything after position 100. A timestamp, a session ID, or a reordered tool list at the top of your prompt breaks caching for the entire request.
• Caches are short-lived. Entries expire after roughly 5 idle minutes. Traffic with gaps longer than the TTL pays the cache write premium over and over without ever collecting a read discount.

Provider by provider

Anthropic (Claude)

Caching is explicit: you mark up to four cache_control breakpoints in the request. Cache writes cost 1.25x list price (5-minute TTL) or 2x (1-hour TTL); reads cost about 10%. The catch: SDKs don't set breakpoints for you. No breakpoint, no cache — many teams pay full price simply because nothing in their stack adds cache_control.

OpenAI (GPT)

Historically automatic for stable prefixes of 1,024+ tokens, with cached input billed at a steep discount. On the GPT-5.6 generation we measured a change: plain SDK traffic stopped getting cross-request prefix hits (0% in our benchmark) — restoring caching requires an explicit cache breakpoint plus a stable prompt_cache_key. Verified live, that took the same workload from 0% to 97.8% prefix hits on steady traffic (S6 cell of our published benchmark).

Google (Gemini)

Two mechanisms: implicit caching (automatic prefix discounts) and explicit context caching, where you create a cache object for a large context and pay a small per-hour storage fee in exchange for much cheaper reads. Great for very large, long-lived contexts; the implicit path behaves like OpenAI's.

xAI (Grok)

Automatic prefix caching with cached-input discounts, OpenAI wire format. Hit rates improve when requests carry a stable conversation/routing hint — an x-grok-conv-id header keeps a conversation's traffic landing on the same cache.

The economics: when caching pays (and when it backfires)

Because writes carry a premium, caching is not automatically free money. The break-even is simple: a 5-minute-TTL write costs 25% extra on the cached tokens, and each read saves ~90%. One hit within the TTL already pays for the write several times over. But if your calls arrive further apart than the TTL, you pay the premium on every call and never collect — hand-tuned caching on sparse traffic can be more expensive than doing nothing. We measured exactly that in our public benchmark: on support-style traffic with 6–9 minute gaps, DIY breakpoints lost money, while keeping the cache warm with tiny pings (each one counted against the savings) came out 67% cheaper than direct.

Rule of thumb: median gap < TTL → set breakpoints and enjoy. Median gap > TTL → either warm the cache through the gaps, switch to a longer TTL, or don't cache — and only measurement tells you which.

Why real-world hit rates disappoint

• TTL expiry — one quiet stretch and the next user pays cold-start price.
• Unstable prefixes — timestamps (“Current time: …”), request IDs, randomized few-shot examples, A/B copy, or tools serialized in nondeterministic order.
• Missing breakpoints — Anthropic traffic with no cache_control at all.
• Sub-minimum prompts — prefixes below the cacheable minimum (e.g. 1,024 tokens) never cache.
• Fleet effects — deploys and model switches cold-start everything at once.

None of these show up as errors. The request succeeds, the answer is fine, and the only symptom is a line item you can't see: most teams assume 60–70% hit rates and measure 20–30%.

Doing it right, automatically

Everything above is automatable, and that's what Caching.ai does: a drop-in proxy that injects breakpoints where they're missing, detects cache-breakers and names the likely cause, keeps prefixes warm through idle gaps exactly as long as the math favors it, picks the right TTL from your real traffic rhythm, and — first of all — shows you the hit rate and wasted spend you actually have. One base-URL swap, 20% of verified savings, free under $5/month, Apache-2.0 core. If you'd rather do it by hand, our playbook is in How to reduce LLM API costs — the checklist is the same either way.