Articles

RAG Interview Questions 2026: Chunking, Retrieval, Reranking, and Evaluation

Practice RAG interview questions on chunking, hybrid retrieval, reranking, evaluation, permissions, latency, freshness, and debugging in 2026.

RAG Interview Questions 2026: Chunking, Retrieval, Reranking, and Evaluation

Quick answer: This 2026 RAG interview guide prepares you to improve a retrieval-augmented generation system by isolating its stages. Be ready to explain how you split source material into useful chunks, retrieve candidates with dense, sparse, or hybrid search, rerank those candidates for precision, and evaluate retrieval separately from the final answer. Strong candidates do more than name components: they define a failure, choose a metric that can reveal it, run a controlled experiment, and discuss quality, latency, cost, freshness, and permissions.

Employer formats, datasets, infrastructure, and scoring criteria vary.

What a focused RAG interview is testing

A focused RAG round is usually about evidence flow: what enters the index, what reaches the model, why the system failed, and how you know a change helped. This guide stays question-led and diagnostic. For a complete ingestion-to-serving architecture, read Designing RAG Architecture at Scale. For broader model, agent, guardrail, and operations decisions, use the Generative AI System Design Interview Questions guide.

Keep four layers distinct:

Layer	Typical design decision	Failure evidence
Chunking	Boundary, size, overlap, hierarchy, metadata	Relevant facts are split, diluted, or missing
Retrieval	Dense, sparse, hybrid, filters, query transformation	Relevant chunks never enter the candidate set
Reranking	Rank fusion, cross-encoder, or model-based scoring	Good candidates exist but appear too low
Evaluation	Dataset, labels, metrics, slices, release gate	A change looks better overall while important cases regress

Use a stage → failure → metric → experiment loop

First name the stage you suspect. Then state the observable failure, select a metric that detects it, and propose one experiment that changes a single variable. It prevents the common mistake of changing prompts when the required evidence never reached the model.

Chunking interview questions

1. What is RAG, and when would you use it?

Strong answer signals: Define RAG as retrieving external evidence at request time and supplying selected context to a generator. Use it when answers depend on private, changing, or source-backed information that should not live only in model parameters. It can improve grounding and support citations, but retrieval does not guarantee truth.

Also name cases where RAG may be unnecessary: stable tasks already handled reliably by the model, deterministic lookups better served by an API or database query, or workloads whose latency and operational cost outweigh the benefit. Start from the user task and evidence requirement, not from a vector database.

2. How would you choose a chunking strategy?

Strong answer signals: Begin with document structure and the questions users ask. Microsoft's current RAG chunking guidance treats document type and structure as starting points, not one fixed token count. Preserve headings, paragraphs, table rows, code units, speaker turns, or other semantic boundaries when they carry meaning. Use overlap only when facts regularly cross boundaries; too much overlap increases duplication and can crowd the context.

Compare alternatives on labeled queries. Measure whether the smallest self-contained passage containing the answer is retrievable, then inspect answer quality, index size, latency, and duplication. For long documents, consider parent-child retrieval: search smaller child chunks, then provide a larger parent section when surrounding context is needed.

3. How can you tell that chunking is the problem?

Strong answer signals: Inspect retrieved text, not only similarity scores. Warning signs include incomplete sentences, a heading separated from its content, tables detached from labels, repeated near-identical chunks, or a fact split across two candidates. Create examples that require cross-boundary context and label the supporting spans.

The experiment should hold the embedding model and retrieval settings constant while changing boundaries, size, or overlap. Compare retrieval recall and downstream answer quality by document type. If the correct intact passage is already retrieved but the answer is wrong, chunking may not be the limiting stage.

Retrieval and permissions interview questions

4. When would you use dense, sparse, or hybrid retrieval?

Strong answer signals: Sparse retrieval is strong for exact terms, identifiers, rare names, and lexical matches. Dense retrieval can match paraphrases and semantic similarity. Hybrid retrieval combines both candidate signals and is useful when queries mix natural language with exact product codes, policy names, or error messages.

Do not claim one method always wins. Build representative queries, including misspellings, acronyms, exact strings, and paraphrases. Compare recall at a candidate cutoff, latency, and operational complexity. Microsoft's current RAG retrieval guidance describes hybrid search and rank fusion as workload-dependent choices.

5. How do metadata filters, freshness, and permissions change retrieval?

Strong answer signals: Treat authorization as a hard constraint, not a ranking hint. Apply tenant, user, region, document-status, and entitlement filters before exposing content; recheck permissions when evidence is used. Store source identity and version metadata so results can be traced and stale entries replaced.

Test both relevance and isolation. Include cross-tenant attempts, revoked access, newly published documents, deleted documents, and conflicting versions. A highly relevant unauthorized chunk is still a severe failure. If filters reduce recall, diagnose indexing or metadata coverage rather than relaxing access controls.

6. When should you rewrite or decompose a query?

Strong answer signals: Rewrite when vocabulary mismatch or conversational context hides the searchable intent. Decompose when one question contains multiple facts that require different evidence. Preserve the original query for traceability, place limits on generated subqueries, and avoid transformations that silently change names, dates, or constraints.

Evaluate the transformation against a no-rewrite baseline. Track whether relevant evidence enters the candidate set, whether unrelated evidence increases, and what the extra calls cost. Include ambiguous and adversarial queries; a sophisticated rewrite that improves average recall but changes user intent is not a win.

Reranking interview questions

7. Why rerank, and where does it belong?

Strong answer signals: First-stage retrieval is optimized to find a broad candidate set quickly. Reranking spends more computation to order that smaller set by relevance before context assembly. Microsoft's retrieval guidance frames that as a recall-first, precision-second trade-off and recommends measuring both relevance and latency. This can improve precision without running an expensive scorer over the full corpus.

Name the trade-off: retrieving more candidates can raise recall, while reranking them adds latency and cost. Benchmark the reranker on labeled query-passage pairs and measure end-to-end impact. Do not assume a higher reranker score is an absolute confidence value; thresholds and cutoffs need empirical calibration on the target workload.

8. How would you compare RRF, a cross-encoder, and an LLM reranker?

Strong answer signals: Reciprocal rank fusion (RRF) combines positions from multiple ranked lists, such as dense and keyword search; it does not deeply interpret each query-passage pair. A cross-encoder scores the query and candidate together, often improving semantic precision at additional compute cost. An LLM reranker can apply richer instructions, but may add more latency, expense, and output variability.

Choose with the same labeled candidate sets. Compare relevance metrics, tail latency, throughput, cost, and failure slices. A staged design can fuse inexpensive retrieval lists, apply a cross-encoder to the reduced set, and reserve an LLM for a narrow high-value case—but only if each stage earns its complexity.

Evaluation and debugging interview questions

9. How do you evaluate retrieval separately from generation?

Strong answer signals: Label which passages support each query. Evaluate retrieval with metrics such as recall@k, precision@k, mean reciprocal rank (MRR), or normalized discounted cumulative gain (nDCG), chosen to match whether coverage, first-hit position, or graded relevance matters. Then evaluate the answer for correctness, support, citation accuracy, completeness, and appropriate refusal. AWS's RAG evaluation documentation likewise separates retrieval context metrics from generated-response and citation metrics.

This separation localizes failure. If supporting evidence never appears, work on indexing, filters, queries, or ranking. If it appears clearly but the answer contradicts it, investigate context assembly, instructions, and generation. Report latency and cost alongside quality rather than hiding them in a separate dashboard.

10. How would you build a useful RAG evaluation set?

Strong answer signals: Sample real user intents and add targeted cases: easy lookups, paraphrases, multi-hop questions, exact identifiers, ambiguous requests, unanswerable questions, stale or conflicting sources, permission boundaries, and long-tail content. Keep a held-out set and turn confirmed production failures into regression cases.

Document label provenance and expected evidence, not merely a reference answer. Calibrate subjective judgments with human reviewers and use deterministic checks where possible. OpenAI's evaluation guidance recommends task-specific, production-like tests and continuous evaluation; it does not supply one universal RAG score or release threshold.

11. The expected document exists, but the answer is wrong. How do you debug it?

Strong answer signals: Trace one failing request through ingestion status, permissions, chunk boundaries, query transformation, retrieved IDs, candidate ranks, reranked order, context truncation, and final output. Identify the first stage where expected evidence disappears or becomes unusable.

Then apply the diagnostic loop: stage → failure → metric → experiment. For example, if the supporting chunk is retrieved at rank 18 but only five chunks reach the model, measure first-hit rank and test reranking while keeping the generator fixed. Add the case to a regression set. Do not change chunking, prompts, embeddings, and top-k simultaneously.

12. Offline quality improved, but production latency or user outcomes worsened. What next?

Strong answer signals: Confirm that offline traffic, labels, and metrics represent production. Slice by query type, source, tenant, language, and answerability. Inspect tail latency, timeout rates, empty retrievals, click or task outcomes, escalation, and cost per successful answer.

Test the smallest correction: reduce candidates, route only difficult queries to an expensive reranker, cache where freshness permits, shorten context, or revert the change. Use a canary with explicit abort criteria. A senior answer explains which quality gain is worth which operational cost and how production evidence can overturn an offline result.

How to prepare for a RAG interview

Build one small baseline and preserve its traces: original query, transformation, retrieved IDs, ranks, selected context, answer, citations, latency, and version information. Create ten to twenty labeled questions across both normal and failure cases. Change one variable at a time and explain the result aloud.

Practice two answer depths: a 90-second design with assumptions and a deeper diagnosis with metrics and experiments. Memorizing vendor defaults is less valuable than explaining why a cutoff, chunk size, or reranker suits a specific workload.

Frequently asked questions

What RAG topics should I study for a 2026 interview?

Study chunking, embeddings, sparse and dense retrieval, hybrid search, metadata filters, query transformation, reranking, context assembly, retrieval and answer evaluation, permissions, freshness, latency, cost, and production debugging. Depth should match the role.

Which retrieval metric is most important?

There is no universal winner. Recall@k helps when missing evidence is costly; precision@k helps when irrelevant context is damaging; MRR emphasizes the first relevant hit; nDCG handles graded relevance and rank. Tie the metric to the product's failure cost.

Is RAG evaluation the same as checking answer accuracy?

No. Retrieval evaluation asks whether the right evidence was found and ranked. Answer evaluation asks whether the model used that evidence correctly, completely, and safely. Measuring both reveals where to intervene.

Should every RAG system use a reranker?

No. A reranker must deliver enough relevance improvement to justify added latency, cost, and complexity. Simple retrieval may be sufficient for narrow, clean corpora; harder or mixed-intent workloads may benefit more.

How should I discuss chunk size in an interview?

Avoid presenting one token count as a rule. Explain the document structure and query pattern, establish a fixed-size baseline, test alternatives, and compare retrieval, answer quality, duplication, index cost, and latency on representative cases.

Final interview checklist

Before finishing a RAG answer, state the user task, evidence boundary, suspected stage, observable failure, metric, experiment, permission model, and quality-latency-cost trade-off. That reasoning is more persuasive than a catalog of tools.

Last reviewed: August 30, 2026. Product capabilities and interview formats can change.