Articles
Semantic Cache Interview Questions: Similarity Thresholds, Freshness, and Wrong Reuse
Practice semantic cache interview questions on similarity thresholds, freshness, permissions, invalidation races, and wrong reuse with worked examples.
A support assistant answers “Can I return this item after 30 days?” by reusing a response to “Can I return this item within 30 days?” The sentences are close in meaning space. Their correct answers may be opposite. A faster response is not a successful cache hit when it answers the wrong question.
The central semantic-cache interview answer: vector similarity proposes a reusable response; it does not prove that reuse is valid. Check exact scope, current source state, and answer compatibility before serving a cached result. Then evaluate wrong reuse alongside latency and cost savings.
This guide separates official implementation facts from proposed design reasoning. All scores, traffic counts, and costs below are synthetic exercises. It makes no candidate-report or company interview-frequency claims. Start with Design an Enterprise Research Assistant with Verifiable Citations and explain what happens when a source changes after an answer is cached.
How is semantic caching different from prompt caching?
Official fact: Anthropic's prompt-caching documentation describes reuse of previously processed prompt prefixes. The model still generates a response for the request. This differs from returning a previously generated answer because the new question resembles an old one. Claude prompt caching
A response cache avoids some downstream work by serving a stored result. An exact response cache uses an explicit identity for the request and relevant context. A semantic response cache adds a similarity-based lookup to find candidates across different wording.
Interview reasoning: these mechanisms can coexist, but their correctness conditions differ. Reusing computation for an unchanged prefix does not establish that a prior final answer fits a new request. Similarly, an exact text match is insufficient when the user's permissions, conversation, source documents, or required output format changed.
Define the cached object first. Is it an embedding, a retrieved document set, a tool result, or a final response? A cache of retrieved candidates still allows fresh answer generation; a final-answer cache commits to much more. Discuss the layer whose work you actually intend to skip.
What does a similarity threshold mean?
Official implementation example: Redis's node-redis semantic-cache guide uses cosine distance, where smaller values indicate closer vectors. Its example retrieves a candidate within metadata filters, then applies the distance threshold in application code. Finding a nearest neighbor is not itself an accepted cache hit. Redis semantic-cache lookup
Design inference: before discussing a threshold such as 0.1 or 0.9, specify the metric, direction, embedding model, preprocessing, and index configuration. A similarity score and a distance score can move in opposite directions. Neither should be interpreted automatically as the probability that an answer is correct.
For the exercises here, use an invented similarity score where larger means closer. This is a teaching convention, not a Redis default. The important distinction is between “these requests are related” and “the same response satisfies both requests under current conditions.”
Small changes in negation, dates, quantities, product versions, or units can alter that second judgment. Replacing an embedding model can also change score distributions. Recalibrate the reuse policy rather than assuming that an old threshold preserves the same error rate.
Which fields must be exact even when wording can vary?
Official fact: Azure API Management exposes vary-by expressions to partition semantic caching and recommends user or group identifiers to control cross-user cache access. Azure semantic-cache policy
Proposed design: construct an eligibility namespace before approximate matching. Depending on the application, it includes tenant, authorization scope, locale, task type, source version, prompt-policy version, model configuration, and output schema. Choose fields because changing them can change the valid response, not because a longer key looks safer.
Tenant isolation alone is not sufficient when two people in one tenant have different document access. Conversely, putting a user ID in every key may unnecessarily prevent reuse of genuinely public answers. Define the narrowest sharing scope that preserves the application's access rules, then enforce current authorization before delivery.
Use this original false-hit test set. The scores are deliberately invented to show why similarity cannot replace exact checks.
| Candidate pair | Reuse decision |
|---|---|
| “Return within 30 days” / “Return after 30 days,” score 0.98 | Reject: the time condition changes. |
| Same question, different private tenant, score 1.00 | Reject: the sharing scope differs. |
| Same question, policy revision 7 versus 8, score 1.00 | Reject or revalidate against revision 8. |
| “Summarize in French” / “Summarize in English,” score 0.97 | Reject: the output requirement differs. |
| Two paraphrases of one public, unchanged FAQ, score 0.96 | Candidate for reuse after quality checks. |
A score of 1.00 does not override a failed gate. If the implementation simply searches globally and returns the highest-scoring answer, the most relevant private result may be exactly the result it must not expose.
How would you choose a threshold without chasing hit rate?
Create labeled request-and-candidate pairs that answer the actual reuse question. Include ordinary paraphrases and hard negatives such as the table above. Labels should evaluate whether the stored answer is acceptable for the new request, not merely whether the prompts discuss the same subject.
Here is a synthetic policy comparison over 1,000 eligible requests. A stricter threshold accepts 200 cached responses, of which 198 are correct. A looser threshold accepts 400, of which 380 are correct.
| Mode | Precision | Errors |
|---|---|---|
| Strict | 99% | 2 |
| Loose | 95% | 20 |
The looser policy doubles accepted hits while multiplying wrong reuse by ten. Neither hit rate alone nor an isolated precision percentage tells you whether that trade-off is acceptable.
For a hand calculation, suppose every accepted hit saves one unit of generation cost, and each wrong reuse incurs a penalty of 50 units. Ignoring lookup costs for the moment, the strict policy's net utility is 200 − 2×50 = 100 units. The loose policy's is 400 − 20×50 = −600 units. These are invented utility units, not vendor prices or a claim that every error has the same impact.
In a real evaluation, separate error severity and treat some conditions as hard constraints rather than priced trade-offs. A permission violation should not become acceptable because enough other requests saved money. Tune on development data, then validate on held-out queries and relevant language, tenant, and task slices.
Does a TTL guarantee that an answer is fresh?
No. Official API fact: RedisVL supports default and per-entry time-to-live settings, along with metadata and filter fields. Expiration controls entry lifetime; it does not inspect whether the answer's underlying source remains true. RedisVL cache API
Interview reasoning: an answer generated one second ago can already be stale if a source changed immediately afterward. An answer generated yesterday may still be valid if it summarizes an immutable document version. Freshness depends on dependencies and the application's consistency requirement, not just age.
Store source identifiers and versions alongside the answer when they are known. A coarse corpus-version namespace is simple but can invalidate many unrelated entries. Fine-grained dependency tracking preserves more hits but requires reliable change propagation and a way to identify every affected answer.
A sliding TTL deserves special attention. If each hit refreshes expiry, a frequently requested stale answer can survive indefinitely unless absolute age or source-version checks constrain it. Distinguish last access, creation time, source verification time, and expiration in both the implementation and the dashboard.
For current inventory, changing entitlements, or another rapidly mutable answer, fresh retrieval or a narrow exact tool-result cache may be preferable. “Do not cache this response class” is a legitimate design decision when correctness costs exceed the work saved.
What race can repopulate the cache after invalidation?
Consider this original concurrency scenario. Request A starts generating from source revision 7. The source advances to revision 8, and an invalidation worker removes revision-7 cache entries. Request A then finishes and writes its old answer back into an unversioned namespace.
The invalidation succeeded, yet the stale answer returned. The bug is the write path's failure to preserve the version that justified the answer.
One approach is to write into a revision-7 namespace and let current readers search only revision 8. Another is to check the source version before insertion and reject a stale write. If the version can change between that check and the write, the check must be part of an appropriate conditional or atomic operation. A check followed by an unrelated write leaves the race open.
Do not confuse this with request coalescing. Coalescing can prevent identical misses from launching duplicate generation, but it does not make a result valid after its dependencies change. The coalescing identity needs the same scope and version discipline as the cache identity.
If invalidation events arrive late or out of order, use version comparison rather than assuming arrival order reflects source order. Include this sequence in tests; a happy-path TTL test will not expose it.
What else belongs in the cache admission policy?
A retrieved answer should have been eligible for storage in the first place. Proposed admission rules: avoid caching partial streams, transient errors, responses that failed output validation, and results whose dependencies cannot meet the chosen freshness contract. Decide separately whether a stable refusal or a negative result is cacheable; reusing a temporary “not found” answer after new content appears can hide valid data.
Conversation-dependent requests need enough context to preserve meaning. “What about the second one?” cannot safely match across unrelated conversations just because the latest message is identical. Include resolved task context or bypass final-answer reuse when it cannot be represented reliably.
Prevent untrusted input from writing directly into a shared trusted-answer cache. Record provenance and validation status, and avoid promoting a cached answer into fresh evidence merely because it has been reused many times. Popularity is not a correctness check.
Limit what you store and log. Embeddings and prompts can carry sensitive information even when they are not displayed as plain text in a dashboard. Apply the same access, retention, and deletion requirements to the cache that apply to its source data.
How would you prove that the cache helps production?
Measure the whole added path: embedding latency, vector lookup, eligibility checks, answer validation, and miss fallback. A miss may cost more than the original uncached request because it performs both the lookup and generation. The relevant improvement depends on traffic mix and accepted-hit frequency.
Report accepted-hit rate, audited wrong-reuse rate, miss latency, hit latency, expiration and version-rejection rates, and downstream work avoided. Separate semantic-neighbor retrieval from final acceptance so a high retrieval count does not masquerade as useful reuse.
A shadow rollout can propose cache hits without serving them, then compare candidates against current evidence or reviewed answers. Fresh generation is not automatically a perfect oracle; use task-specific checks and human judgment where needed. Sample near-threshold and high-impact cases deliberately, and preserve sampling weights when estimating population error.
Track embedding, policy, source, and schema versions through releases. Test cache outage behavior and bound lookup time so an optional optimization cannot indefinitely block the main response. Roll back a policy that improves average latency while worsening answer validity beyond the accepted limit.