# Semantic Response Cache in Front of LLM Calls

> Reuse prior answers for near-duplicate prompts using embedding similarity with a strict threshold and TTL.

Category: Multi-Model Routing  
Tags: caching, embeddings, cost, latency  
Setup time (estimate): 1-2 hours  
Estimated cost: One embedding per request plus vector storage; saves generation cost on hits.  
Last verified: not yet independently verified

## Steps

1. Normalize the prompt (trim, lowercase for matching only) and include model + system prompt hash in the cache key scope.
2. Exact-match lookup first (hash). On miss, embed the prompt.
3. Nearest-neighbor search within the same scope; accept only if cosine similarity ≥ threshold.
4. On miss, call the model and store `(embedding, prompt, response, created_at)`.
5. Expire entries by TTL; never cache responses that depend on time or user-specific data.

## Config

```yaml
semantic_cache:
  exact_match_first: true
  similarity_threshold: 0.95   # start strict; lower only after reviewing false hits
  ttl_hours: 24
  scope_by: [model, system_prompt_hash, tool_set_hash]
  skip_if: [has_tools_with_side_effects, temperature_gt_0_5, user_specific_context]
```

LiteLLM and GPTCache both offer built-in semantic caching if you prefer not to build it.

## Failure modes

- Too-low similarity threshold returns wrong cached answers.
- Cache not scoped by model and system prompt serves stale behavior.
- Personal or time-sensitive answers cached across users; exclude them.