Skip to main content
Subscribe
AI & Agentic

Semantic Caching for LLM Apps: Dual-Layer Architecture Cuts Bills 73%

October 11, 2026 · by Nishil Bhave · 14 min read
A dual-layer semantic cache pipeline - query, L1 hash check, L2 HNSW vector lookup, threshold gate - branching into a 67% cache hit versus an LLM call, cutting the bill 73%

Why Is Your LLM Bill Growing Faster Than Your Traffic?

Your LLM bill keeps climbing. You add a feature, traffic doubles, the bill triples. Then someone on the team asks the obvious question: “Why are we paying a frontier model every time a user asks ‘how do I reset my password’ rephrased six different ways?”

That’s the gap semantic caching fills. Not the prompt cache built into OpenAI or Anthropic. That’s a different thing entirely. We’re talking about catching "how do I reset my password" and "forgot password help" as the same question, and serving the cached answer for both.

One documented case study cut LLM API costs from $47,000 to $12,700 monthly, a 73% reduction, by raising cache hit rate from 18% (exact-match only) to 67% (semantic) (VentureBeat, 2026). In a November 2025 evaluation on ElastiCache and Bedrock, AWS measured a 90.3% hit rate with 91.2% answer accuracy at a permissive 0.75 similarity threshold, cutting LLM cost by 86% (Amazon Web Services, 2025).

I’ve been running a semantic cache in production for Growth Engine‘s agent workflows for the past six months. This is the architecture that worked, the one that didn’t, and the threshold tuning matrix I use today.

how agentic AI workflows generate repetitive LLM calls and where caching fits

Key Takeaways

  • A hash-only cache catches ~18% of repeats; semantic caching pushes that to 67%+ on chat-style traffic (VentureBeat, 2026).
  • The right architecture is dual-layer: exact-hash first (sub-ms), HNSW vector lookup second (~25ms), LLM call last.
  • Tune similarity thresholds per query class: about 0.78 for factual lookups, 0.86 for conversational, 0.92 for technical Q&A, 0.95+ for code or legal answers.
  • A semantic cache hit took 27ms versus about 7 seconds for the API call in Percona’s 2026 test, a 250× speedup (Percona, 2026).

What Is Semantic Caching for LLMs?

Semantic caching stores LLM responses keyed by the meaning of the request, not the exact byte-for-byte string. Two questions with different wording but the same intent, like “How do I reset my password?” and “I forgot my password, what now?”, return the same cached answer, because their embeddings sit close together in vector space.

The mechanism is straightforward. You embed the incoming prompt, run a similarity search against previously cached prompts, and if any cached entry sits within your similarity threshold, you serve its response. If nothing matches, you call the LLM, store the new pair, and return the fresh answer.

Analysis of 100,000 production LLM queries found only 18% were exact duplicates of earlier queries, while another 47% were semantically similar: same intent, different wording (VentureBeat, 2026). That 47% is the entire reason this technique matters. Exact-match caching ignores it. Semantic caching captures it.

Production LLM query analysis showing 18% exact duplicates, 47% semantically similar, and 35% unique queries

Source: VentureBeat (2026) – 65% of production LLM queries are cacheable, but only semantic caching catches the 47% rephrased middle.

The real cost win shows up at scale. In Percona’s February 2026 test, a cached response came back in 27 milliseconds (23ms embedding, 2ms vector search, 1ms fetch) versus about 7 seconds for the LLM API call, a 250× speedup (Percona, 2026). For an agent that makes ten LLM calls per user action, that’s the difference between a usable product and a slideshow.

why agentic workflows generate repetitive prompts and amplify caching value

Why Doesn’t Prompt Caching Solve This?

Prompt caching and semantic caching solve different problems. Prompt caching, what OpenAI and Anthropic ship, caches the prefix of a prompt at the model layer. Semantic caching skips the model entirely.

Anthropic bills cache reads at 0.1× the base input price when you mark sections of a prompt as cacheable with cache_control, and even less on its top models: 0.05× on Opus 5.5 and 0.025× on Fable 5.1. Cache writes cost 1.25× base input for the 5-minute cache and 2× for the 1-hour cache (Anthropic docs, 2026). OpenAI’s GPT-6.1 Sol charges $0.10 per million cached input tokens against $2.00 standard, with cache writes at $2.50 (OpenAI, 2026). Either way, the cached portion of your input gets roughly a 90-95% discount.

But notice what prompt caching doesn’t do. It still calls the model. It still pays for output tokens. It still incurs network latency to OpenAI or Anthropic. And critically, it requires the cached prefix to be byte-identical. Change one word in your system prompt and the prefix cache misses entirely.

Here’s the practical decomposition I use:

From production: We use both. Prompt caching handles our 4,000-token system prompt and tool schema across every agent invocation, that’s a flat ~85% discount on input tokens. Semantic caching sits in front of the whole stack and catches the 30-40% of user questions that are rephrasings of prior questions. The two layers compose; they don’t compete.

The mistake teams make is treating these as alternatives. They’re not. Prompt caching cuts the per-call cost. Semantic caching cuts the number of calls. Stack them.

how agent loops compound LLM cost without caching

How Does the Dual-Layer Architecture Actually Work?

The dual-layer pattern runs three checks in sequence: exact-hash lookup, semantic vector lookup, then the LLM. Each layer catches a different class of repetition, and the order matters because each layer is exponentially more expensive than the one before it.

Here’s the request path I run in production:

1. Normalize prompt (lowercase, trim, strip safe whitespace)
2. SHA-256 hash → Redis GET → if hit, return (~0.5ms)
3. Embed prompt with text-embedding-3-small (~22ms)
4. HNSW vector search in Redis/Qdrant (~3ms)
5. If similarity ≥ class_threshold → return cached response (~25ms total)
6. Else → LLM call → store {hash, embedding, response} → return (~6s+)

The first layer is just a SHA-256 of the normalized prompt. It catches the 18% exact-match traffic in under a millisecond, with zero risk of a wrong answer. Always run hash before embedding. Hash hits cost nothing. Embedding costs an API call every time. It’s fractions of a cent per call, but it adds up at scale and it adds latency.

The second layer is the semantic part. You embed the prompt, then run an approximate nearest-neighbor search using HNSW (Hierarchical Navigable Small World) against previously-stored embeddings. On a billion-vector benchmark, Redis reported 90% precision at about 200ms median latency, or 95% precision at about 1.3 seconds with HNSW tuned for accuracy (Redis, 2026). At the scale most teams operate (under 10 million cached entries), HNSW returns in 2–5ms in my setup.

Latency comparison showing exact hash hit at about 0.5ms, semantic cache hit at about 27ms, and uncached LLM call at about 7000ms

Sources: Percona (2026) for the semantic hit and API call; the hash-hit figure is from my own Redis setup. Each layer is far cheaper than the next, which is why ordering matters.

Why not skip straight to vector search? Because embedding the prompt costs you an embedding API round-trip even on a hash hit. For high-traffic systems with repeated identical queries (think “what’s my balance?” hammered every page load), the hash layer alone can shave 5–10% off your embedding costs. It’s free latency.

Gateways have converged on the same split. LiteLLM, for example, ships a plain redis exact-match cache alongside redis-semantic, qdrant-semantic, and valkey-semantic cache types, each with a configurable similarity_threshold (LiteLLM, 2026). The architecture has converged because the math is unforgiving.

A dark server rack with rows of glowing equipment indicators representing the cache infrastructure layers in production

How Do You Tune the Similarity Threshold?

Threshold tuning is where most teams get this wrong. They pick one number, usually 0.85 or 0.90, apply it everywhere, and either bleed accuracy on factual queries or miss obvious paraphrases on chat traffic. The fix is to tune by query class, not globally.

Cosine similarity is the standard metric. A score of 1.0 means the two embeddings are identical; 0.0 means orthogonal. In practice, well-formed prompts in the same domain tend to cluster between 0.75 and 0.99. The question is where you draw the cache-hit line.

Published guidance clusters high. Portkey’s April 2026 guide recommends starting between 0.90 and 0.95, and says Portkey itself starts around 0.95, then backtests on roughly 5,000 real queries until accuracy stays above 99%, based on more than 250 million cache requests (Portkey, 2026). The VentureBeat team landed on different numbers per query type: 0.94 for FAQs, 0.88 for product searches, 0.92 for support queries, and 0.97 for transactional queries (VentureBeat, 2026). At 0.85, “How do I reset my password?” can match “How do I change my email?”, since both are account-management questions in similar prose. At 0.98, you’re effectively back to exact matching.

The right number also depends on your data. In AWS’s evaluation, accuracy barely moved between thresholds: 92.6% at 0.95 and 91.2% at 0.75, while hit rate jumped from 56.0% to 90.3% (Amazon Web Services, 2025). Their chatbot queries were forgiving. Yours may not be.

But any single global number is a lossy approximation. Here’s the per-class matrix I use today:

Similarity threshold tuning matrix showing recommended thresholds by query class from 0.78 for factual lookups to 0.97 for code generation

Source: My production thresholds at Growth Engine (2026), not a vendor benchmark. Threshold tightens as the cost of a wrong cached answer rises.

The pattern is straightforward. The riskier a wrong answer is, the higher the threshold. A wrong factual lookup (“what year was the Eiffel Tower built?”) is mildly annoying. A wrong code generation answer is a bug. A wrong legal or medical answer is a lawsuit. Tune accordingly.

Our finding: When we A/B tested a global 0.90 threshold versus per-class thresholds at Growth Engine, the per-class config delivered a 22% higher hit rate at the same false-positive rate. The trick was classifying the prompt with a small fast LLM (GPT-4o-mini when we built it; any current small model such as Claude Haiku 4.5 does the job) before the embedding step, then routing to the right threshold pool. Cost of the classifier: $0.0001 per call. Win on cache hit rate: dramatically more than that.

The other knob people miss is embedding model choice. text-embedding-3-small (1536 dimensions) and text-embedding-3-large (3072 dimensions) produce different similarity distributions. A 0.92 with small is roughly equivalent to 0.94 with large. Re-tune when you switch models. Don’t assume the threshold transfers.

why code-generation cache hits demand higher thresholds than chat

Which Vector Store Should You Pick?

For semantic cache duty specifically, you need three things from your vector store: low p99 latency, mutate-while-reading support, and an HNSW or equivalent ANN index. Most production teams in 2026 land on one of three choices.

Redis with vector search. This is the path of least resistance if you already run Redis for hot data. LiteLLM supports Redis for both the exact-match and the semantic layer, so one infrastructure component covers both layers of the cache. Redis’s query engine handles HNSW natively and survives the bandwidth most LLM gateways throw at it.

Qdrant. Open-source, written in Rust, designed specifically as a vector database. Qdrant tends to win on raw throughput per dollar when you’re caching tens of millions of entries. LiteLLM supports it as a semantic cache backend too, which makes it the natural next step for teams that outgrow Redis.

Postgres with pgvector. Not the fastest option, but the simplest if your team already lives in Postgres. With pgvector + HNSW indexing, you get cache reads in 5–15ms at moderate scale, plus the ability to JOIN cache entries with user metadata for per-tenant isolation. It’s also the easiest to back up.

The decision rule I use: start with Redis if you already have it, switch to Qdrant when you cross 10 million cached entries, use pgvector if your dominant constraint is operational simplicity. Don’t over-engineer this. The vector store is rarely the bottleneck, the embedding API call is.

ACID guarantees and why they matter for cache write atomicity

How Do You Handle Cache Invalidation and Drift?

Cache invalidation is the hard part. Your LLM responses go stale when the underlying knowledge changes: you push new docs, the product behavior shifts, a fact about the world updates. A six-month-old cached answer to “what’s our refund policy?” is worse than no cache at all.

Three invalidation strategies actually work in production:

In practice: I default to TTL-based eviction with a hard 24-hour ceiling, tag-based invalidation on top of TTL for any RAG response, and drift detection running at 2% sample rate as a safety net. Three layers, but each one is doing distinct work. The TTL handles the boring 80%. The tags handle the “we just shipped a doc update.” The drift detection handles everything else.

There’s one more failure mode worth naming: cache poisoning. If your LLM ever returns a hallucinated or unsafe answer and you cache it, every subsequent semantic match serves the same broken response. Validate before you cache. At minimum, run safety filters on the response before storing. For high-stakes domains, run a second model as a judge. The cost of judging beats the cost of serving a cached hallucination ten thousand times.

using a judge model to gate cache writes

What Are the Real Cost Numbers?

The published cost reductions cluster in a tight band: 40–86% across documented production deployments. The variance comes almost entirely from query mix and threshold tuning, not from the choice of cache architecture.

Here’s the spread from public benchmarks:

Savings reported in semantic caching benchmarks: VentureBeat 73% cost reduction, AWS 86% cost reduction, GPT Semantic Cache 68.8% fewer API calls, and SCALM 77% relative token-savings gain over GPTCache

Sources: VentureBeat (2026), AWS (2025), arXiv 2411.05276 (2024), arXiv 2406.00025 (2024). The metrics differ (cost, API calls, relative token savings), but all four land between roughly 65% and 86%.

Don’t read these as a promise. Read them as a range. Your hit rate depends on three things: how repetitive your traffic actually is, how aggressively you’ve tuned your thresholds, and whether you’ve stacked the L1 hash layer in front of the L2 semantic layer. A well-tuned dual-layer cache on a chat-style workload should land somewhere between 60% and 80% hit rate. A poorly-tuned single-layer setup will sit at 20–30% and you’ll wonder why everyone else is reporting bigger numbers.

The math that closes the deal for finance teams: Anthropic’s prompt caching gives you a 90% discount on input tokens for cached prefixes on most models, and more on Opus 5.5 and Fable 5.1 (Anthropic docs, 2026). Stack semantic caching on top, and on a 70% hit rate you’re paying for ~30% of calls at ~30% of input cost, call it 9% of your uncached bill on input, plus 30% on output. Total spend lands around 20–25% of the uncached baseline. That’s where the 73–86% reductions come from. It’s not magic. It’s compounding two discounts.

Frequently Asked Questions

What’s the difference between semantic caching and prompt caching?

Prompt caching (KV cache) reuses model attention vectors for a fixed prompt prefix, saves input token cost but still calls the LLM. Semantic caching reuses the entire response when a new request is semantically similar to a cached one, skips the LLM call entirely. Anthropic’s prompt cache bills cache reads at 0.1× base input on most models (Anthropic docs, 2026); a well-tuned semantic cache can eliminate 60–80% of calls outright. Use both.

What similarity threshold should I use for semantic caching?

There’s no universal answer; it depends on query class. In my setup, factual lookups work at 0.78–0.82, conversational chat at 0.86–0.90, technical Q&A or RAG at 0.92, and code generation, legal, or medical at 0.95–0.97. If you have no data yet, Portkey’s advice to start between 0.90 and 0.95 and backtest on real traffic is a safe default (Portkey, 2026). Tune per class, not globally.

Should I use Redis or a dedicated vector database for semantic caching?

Redis with the RediSearch module is the right default for under 10 million cached entries, you reuse existing infrastructure for both the hash layer and the vector layer. Switch to Qdrant or another dedicated vector DB above 10M entries when throughput-per-dollar matters more than operational simplicity (LiteLLM, 2026). Pgvector works if Postgres is already your home.

How do I prevent stale or wrong answers from getting cached?

Three layers: TTL-based eviction (24 hours for chat, 1 hour for live-data queries), tag-based invalidation tied to source documents, and drift detection that replays a 2% sample of cache entries through the LLM and evicts on material divergence. Always run safety filters before storing, caching a hallucination means serving it ten thousand times.

Does semantic caching work for streaming LLM responses?

Yes, but you need to accumulate the full response on first generation, store it, and replay it on cache hit. If you’re building it yourself, buffer the stream while writing to cache, then replay it in chunks on subsequent hits to preserve the streaming UX. Check your gateway’s docs before assuming it does this for you. You lose nothing on the user-facing side and still get the full cost benefit.

Conclusion

Semantic caching is one of the few infrastructure decisions in 2026 where the cost math is simple, the architecture is converged, and the upside is large. A dual-layer cache with class-tuned thresholds drops LLM spend by 60–80% on chat-style traffic, with cache hits returning in tens of milliseconds instead of seconds.

The architecture is now table stakes. Major LLM gateways, including Portkey, LiteLLM, and Kong AI Gateway, offer semantic caching out of the box. If you’re rolling your own, the rules are: hash before embed, embed before LLM, tune thresholds per query class, invalidate aggressively, and validate responses before you cache them.

Most teams underestimate how much of their LLM traffic is rephrased repeats. The 47% middle band of “semantically similar” queries is invisible to exact-match caching and untouched by KV cache. Catch it, and the bill stops growing faster than your traffic.

next, see how agentic workflows compound caching value across multi-step agent loops

Written by Nishil Bhave

Builder, maker, and tech writer at MakeToCreate.

Website All posts →

Never miss a post

Get the latest tech insights delivered to your inbox. No spam, unsubscribe anytime.

By subscribing, you agree to our Privacy Policy and Beehiiv's Terms.

Related Posts