AI COST OPTIMIZATION
Your cache goes cold.
Your bill goes 10×.
Providers discount repeated prompts by 90% — while the cache stays warm. Caching.ai is a drop-in proxy that keeps it warm, so the discount actually lands on your bill. One base-URL swap.
One base-URL swap · Runs on your own provider keys · Prompts are never stored · Save nothing, pay nothing
THE PROBLEM
You're paying full price for prompts your provider already discounts.
Anthropic, OpenAI, Gemini, and Grok all serve repeated prompt prefixes from cache at a fraction of list price. But that discount only shows up if the cache actually gets hit — and in real traffic, it quietly fails far more than teams expect.
One short idle gap between requests and the cache expires — the next call pays full price again.
A cached token costs a tenth of a fresh one. Every miss on a repeatable prompt is money left on the table.
The share of spend teams can recover just by keeping their cache healthy — no model or code changes.
warm — every repeat reads at ~10% of input price → cold — you silently pay 100% again
We make every miss visible, then eliminate the ones that shouldn't happen.
MEASURED, NOT PROMISED
Up to 90% cheaper than calling direct. Proven on the bill.
10,000+ real, billed API calls — the same workloads run direct, hand-tuned, and through caching.ai. Every cost is the provider's own usage numbers at public list prices, and the method, harness, and raw logs are all open source.
Support traffic with 6–9 minute idle gaps · 12 calls · claude-haiku-4.5
what doing it by hand looks like: the cache expires in every gap, so the write premium makes it pricier than doing nothing
the Cache Warmer held the cache warm through every gap — every ping included in the number
100% = what the direct call paid. Mean of 3 independent runs.
Support-style conversations with idle gaps longer than the cache TTL — net of all warming pings.
Classification batch with a shared prefix, vs the same batch with SDK defaults.
The 5.6 generation dropped cross-request prefix caching for plain SDK traffic (0% hits) — the proxy restores it: 97.8% hit rate on steady traffic.
Faster, not slower
A proxy hop sounds like added latency. Measured, it is the opposite: a cache hit skips the prompt prefill, so first tokens arrive sooner — and the hop itself was smaller than provider noise.
gpt-4o, sparse support traffic (S2), provider-reported timing. Lower is better.
Every number above comes from real, billed API calls — method, fixtures, and raw logs are public, and you can rerun the whole benchmark with your own keys. Where a provider already caches well on its own, the proxy simply passes your traffic through at zero overhead.
2026-07 runs · 10,000+ calls · $80+ at list prices · Anthropic cells: 3 runs each · raw logs public
The economics of warming
Warming a cache on purpose? The math says it's cheaper.
"While I'm working the cache stays alive anyway, and if I leave I just let it die — right?" The better you know cache pricing, the fairer that question. But traffic with 6–50 minute call gaps — support bots, Slack bots, pipelines, agents waiting on human approval — structurally expires a 5-minute cache every single time. And rebuilding costs 1.25× list price while reading a live cache costs 0.1×: one rebuild = 11.5 warming pings.
Sixty minutes of calls ~10 minutes apart — the same shape as the benchmark's sparse-support scenario. Each dot is a request; the band behind it is the 5-minute cache lifetime.
Break-even: 52 minutes
For any absence under 52 minutes, warming always wins. Past that point (~62.5 min after your last request) warming stops itself — not a cent is spent outside the profitable window.
Past 90 minutes, switch strategies
Long holds skip the 4-minute pings: the context is rewritten once as a 1-hour cache entry, cutting hourly upkeep 12×. The upgrade runs only after the old entry has expired — no overlapping spend.
Your real requests are untouched
Your requests always go out with their original cache markers. The pricier 1-hour write is attached to warming pings only, and a per-org daily budget cap quietly stops everything past its limit.
The 67% saving in the benchmark above is exactly that middle-zone traffic (support calls 6–9 minutes apart) — on the same traffic, caching without warming lost 25%. Conversely, leaving for hours? Letting the cache expire is cheaper, and in that window we stop first. Warming isn't a trick: it's cost engineering that computes the break-even and spends only when it wins.
WHO IT'S FOR
Built for teams whose AI bill keeps climbing
AI product teams
Long system prompts, RAG context, tool definitions — the same thousands of tokens resent on every request. That's exactly what caching is for, and exactly what breaks quietly.
- Chatbots & copilots with heavy system prompts
- RAG apps rebuilding identical context
- Multi-tenant SaaS with shared prompt scaffolding
Agent fleet operators
Coding agents and automation fleets idle between tasks — just long enough for the cache to expire before the next burst. The Cache Warmer closes that gap.
- Coding agents (Claude Code, custom harnesses)
- Batch pipelines with think-time gaps
- Overnight autonomous runs
Engineering leaders
You own the AI line item but the invoice hides everything. Get per-model hit rates, verified savings, and the waste number nobody else reports.
- One dashboard across Anthropic, OpenAI, Gemini
- Estimated waste — the metric invoices hide
- Weekly savings report in your inbox
THE PRODUCT
Four ways we cut your bill
See what the cache actually saved you
Finally see the number your invoice hides: how much of your spend the cache actually absorbed — and how much it should have.
- Real hit rate per model and per key, from live traffic
- Verified savings vs. list price, plus estimated waste from misses
- Token counts only — prompt and response bodies are never stored
Find and fix what breaks your cache
Most cache failures are self-inflicted: a timestamp, a random ID, a reordered tool list. Guard finds them and fixes what can be fixed automatically.
- Auto cache_control injection for requests that have none (Anthropic)
- Prefix-stability detection across all four providers
- A dashboard warning with the likely root cause, not just a symptom
Keep the cache warm between calls
Anthropic caches expire after 5 idle minutes. Our engine sends tiny 1-token pings to keep your prefix warm exactly while it's economical — never longer, and only where our own benchmark proved pings pay off.
- Pings cost ~0.1× of a cache rewrite — profitable up to 62.5 minutes of idle
- Hard daily budget per key; every ping is metered on your dashboard
- Opt-in per key, prefix stored encrypted (AES-256-GCM)
- Stepping away? Say “keep my cache warm for 2 hours” in chat — the proxy answers by itself (nothing reaches the AI), pre-warms the conversation you asked from, and holds it warm. Understood in 16 languages
- On Claude Code, one plugin makes it fully automatic: every turn silently renews a 2-hour hold, so lunch never costs you a cold cache again
Cache settings that tune themselves
Every key has a rhythm — how far apart its calls really arrive. Auto-Tune learns yours from live traffic and picks the cheapest cache settings for it, then keeps re-checking as your traffic changes.
- Simulates your last 14 days of real call gaps against every cache option
- Switches TTL and retention only when the math clearly favors it — with the evidence shown on the key
- Every decision is logged: median gap, sample size, projected savings
HOW IT WORKS
One base URL. That's the integration.
Zero code changes — one base-URL swap. Streams pass through byte-for-byte.
Point your SDK at us
Set your provider base URL to our proxy and swap in your Caching.ai key. No code changes.
We optimize every request
Usage is captured from the response stream, cache breakpoints are injected where they're missing, and breakers are flagged.
Watch the savings
Your dashboard shows hit rate, dollars saved, and dollars still being wasted — updated in real time.
# beforeANTHROPIC_BASE_URL=https://api.anthropic.com# after — that's the whole migrationANTHROPIC_BASE_URL=https://proxy.caching.aiANTHROPIC_API_KEY=ck_your_caching_ai_key
WHY US
Built for one job: your cache
Generic LLM gateways route and observe. Calling providers directly leaves the cache to luck. We do the one thing that moves the bill.
| Direct to provider | Generic gateway | Caching.ai | |
|---|---|---|---|
| See wasted spend, not just usage | — | △ | ✓ |
| Cache Warmer between calls | — | — | ✓ |
| Cache-breaker detection with the root cause | — | — | ✓ |
| Automatic cache_control / cache routing | — | △ | ✓ |
| Self-learning cache settings (Auto-Tune) | — | — | ✓ |
| Pay only from verified savings | — | — | ✓ |
✓ included · △ partial · — not offered
OPEN SOURCE
The code is all public. Run it yourself, or let us run it for you.
Caching.ai's core is open source (Apache-2.0). Run the exact same proxy and console on your own infrastructure — or use the cloud and never think about servers.
Both, identical
- Proxy, auto cache_control & cache warming
- Cache-breaker detection & diagnostics
- Dashboard & key console
Cloud only
- Zero ops — no install, no updates, no servers
- Auto-Tune: learns your traffic, picks cache settings for you
- Weekly reports & budget alerts out of the box
- Pay 20% of verified savings — waived under $5/mo
Self-host
- Free forever, on your infrastructure
- One docker compose up
- Ops, updates and backups are on you
Either way, prompt and response bodies are never stored.
DO THE MATH
What would a healthier hit rate save you?
Estimated on Opus-tier input pricing with cache reads at 10% of list price. Most teams sit near 20–30% hit rate without trying; well-kept caches reach 70%+. Actual results depend on your traffic pattern.
TRUST & SECURITY
Your keys and prompts, handled like production secrets
Requests travel on YOUR provider keys, straight to the provider — we never resell models or sit on your data. Bodies pass through byte-for-byte and are never stored; what remains is numbers (token counts, latency, prefix hashes). Want zero third parties? Self-host the exact same code.
Caches can never mix between users
The cache itself lives at the provider, scoped to YOUR API key and account. We never cache responses ourselves or serve one user's cache to another — there is no shared cache to mix. It's open source: verify it in the code.
Your keys stay yours
Provider keys are encrypted at rest with AES-256-GCM, used only to forward your requests, and never logged.
Prompts are never stored
We keep token counts, latency, status codes, and prefix hashes — not your content. The one exception, the Cache Warmer, is opt-in and labeled right on the toggle.
Byte-identical passthrough
Responses stream through unbuffered, byte for byte. Your app sees exactly what the provider sent — analytics happen after the response is on its way.
Leave in one click
Delete your account yourself, anytime. Keys, stored prefixes, and your card are wiped immediately — no email ping-pong.
Every optimization is a switch
Cache injection, warming, TTL, retention — all per-key opt-ins you can flip off at any moment. Nothing is forced on your traffic.
PRICING
Pay only when you save.
20% of your net savings
We measure your verified savings against list price, subtract every warming ping we spent on your behalf, and charge 20% of the remainder after the month closes. You keep 80% of money you were already spending.
Under $5? Waived.
Monthly fees under $5 are simply waived — never carried over. Save nothing, pay nothing. Your dashboard shows the exact running fee all month, so the invoice is never a surprise.
FAQ
Frequently asked questions
Do you store my prompts?
No. By default we store only token counts, model names, latency, status codes, and SHA-256 hashes of prefix blocks. The Cache Warmer is opt-in per key and stores an encrypted copy of your prompt prefix (AES-256-GCM) because re-sending it is how the cache stays warm — the trade-off is stated right on the toggle.
How does the proxy authenticate?
You bring your own provider API keys. They're encrypted at rest, used only to forward your requests, and never logged. Your apps call us with a Caching.ai key instead.
What's the latency overhead?
Streaming responses are piped through byte-for-byte with usage read on the fly — no buffering. Analytics writes happen after your response is already on its way.
Which providers are supported?
All four — analytics and cache-breaker detection everywhere, with optimization where it measurably pays: auto cache_control and the Cache Warmer on Anthropic, and on GPT-5.6+ an explicit cache breakpoint plus a stable prompt_cache_key that restore the prefix caching the new models no longer do on their own (measured 0% → 97.8% prefix hits on steady traffic — S6 cell of our published benchmark). OpenAI, Gemini, and Grok hold their caches upstream — we measured warming pings there and turned them off instead of billing you for them.
What does it cost?
Pricing is performance-based: 20% of what we verifiably save you, charged to your card after each month closes. Fees under $5 are waived. If we save you nothing, you pay nothing.
My provider already caches automatically — why do I need this?
Right — the discount exists. The problem is how often you actually get it. Caches expire after ~5 idle minutes, and a single changed character at the start of your prompt silently breaks them. We keep the cache alive, catch what breaks it, and show you the hit rate you're really getting — most teams find it's far lower than they assumed.
What is Auto-Tune?
It learns each key's real call rhythm — how far apart requests actually arrive — and picks the cheapest cache settings for it: the 5-minute vs 1-hour TTL on Anthropic. OpenAI, Gemini and Grok are managed automatically per model — nothing to configure. It only switches when the math clearly favors it, and shows you the evidence (median gap, sample size, projected savings) on the key.
What happens if Caching.ai is down?
Your requests do pass through us, so an outage on our side would fail your calls — and the fix is the same one-line change you made when you connected: point the base URL back at your provider and you're running direct again. No lock-in, nothing to migrate.