Prompt Caching vs Semantic Caching: Which LLM Cache Do You Actually Need?
Semantic caches return a stored answer for similar questions; prompt caching gets you the provider's 90% discount on repeated prefixes. They solve different problems — here's how to choose (or combine) them.
Search for “LLM caching” and you'll find two families of tools that share a word and almost nothing else. Semantic caching (and its simpler cousin, exact-match response caching) stores answers and replays them. Prompt caching stores nothing on your side at all — it's a provider-side billing discount for re-sent prompt prefixes. Teams regularly adopt one thinking they're getting the other, so here is the clean split.
| Prompt (prefix) caching | Semantic / response caching | |
|---|---|---|
| What is cached | The model's processed prompt prefix, at the provider | Your previous responses, in your infrastructure |
| When it helps | Any request repeating a prefix (system prompt, tools, history) — nearly all agent/chat traffic | Different requests asking the same/similar thing |
| Savings per hit | ~90% off the cached tokens; output still billed | 100% — the model never runs |
| Answer freshness | Always fresh — the model runs every time | Replayed — staleness and mismatch risk |
| Latency effect | Faster first token (prefill skipped) | Near-instant on hit |
| Failure mode | A miss: you lose the discount, never correctness | A false hit: the user gets a wrong or outdated answer |
| Ops burden | None (provider-side); the work is keeping prefixes stable and warm | Embeddings, vector store, thresholds, invalidation |
Why the confusion persists
Most AI gateways advertise “caching” and mean the response kind — it's a natural gateway feature. Providers advertise “prompt caching” and mean the billing kind. Both cut costs, so roundups lump them together. But they answer different questions: response caching asks “have we answered this before?”; prompt caching asks “has the model read this before?” In production traffic the second is true an order of magnitude more often than the first.
When semantic caching is the right call
- High volumes of near-duplicate questions from different users (public FAQ bots, search-style interfaces).
- Answers that stay valid for hours or days, with a clear invalidation story.
- You can tune and monitor a similarity threshold — and eat the occasional false hit.
When prompt caching is the right call
- Agents, copilots and chatbots that resend a large system prompt, tool schema or conversation history every call — i.e. most modern LLM apps.
- Anything where a replayed answer is unacceptable (personalized, stateful, or time-sensitive output).
- Batch jobs sharing one prefix across hundreds of calls — our benchmark measured 89% savings on a 300-call batch.
The catch with prompt caching is operational, not architectural: caches expire after ~5 idle minutes, one unstable token at the top of the prompt silently zeroes the hit rate, and Anthropic requires explicit cache_control breakpoints. That upkeep — measuring the real hit rate, stabilizing prefixes, warming through gaps — is exactly what Caching.ai automates behind a base-URL swap. Mechanics in full: What is prompt caching?
Use both — in the right order
The layers stack: put the response/semantic cache in front (a hit there is free), and let every miss flow to the provider through prompt caching so the prefix is billed at ~10%. Just keep the fallacy straight — a semantic cache does not improve your prefix hit rate, and prompt caching will never deduplicate two users asking the same question. Different caches, different jobs.
Frequently asked questions
Can semantic caching return wrong answers?
Yes — that's its core trade-off. If the similarity threshold is too loose, a user asking 'how do I cancel my subscription?' can receive the cached answer for 'how do I change my subscription?'. Prompt caching has no such failure mode: the model always generates a fresh answer; only the billing of the repeated prefix changes.
Which saves more money, prompt caching or semantic caching?
Per hit, semantic caching saves more (100% — the model never runs, vs ~90% off the cached prefix tokens). Across a real workload, prompt caching usually wins because it applies to nearly every request in prefix-heavy traffic like agents and chatbots, while semantic hits only occur when different users ask sufficiently similar questions.
Can I use prompt caching and semantic caching together?
Yes, and it's the right architecture when both fit: check the semantic/exact cache first (a hit costs nothing), and every miss goes to the model through prompt caching so its prefix is billed at the discounted rate. The two layers are independent and stack cleanly.