Articles
Haystack Interview Questions: Pipeline Connections, Document Updates, and Retrieval Failures
Prepare for Haystack interview questions with typed pipeline sockets, document IDs, duplicate policies, revision cleanup, and retrieval-first debugging.
A Haystack pipeline runs without an exception, but its answer quotes an obsolete policy. A second pipeline fails before retrieval because a component output is connected to the wrong input. These failures require different evidence. Check the component contract, stored document identity, retrieved results, and generator input separately before changing the model or prompt.
Start with Design and Evaluate a Retrieval-Augmented Generation Pipeline. Use the case to explain how a retrieval problem becomes visible before a generated answer obscures it.
Which Haystack version does the interview code use?
Official facts: The current Haystack migration guide describes changes from 2.x to 3.x. AsyncPipeline was merged into Pipeline; synchronous run() executes components sequentially, while asynchronous methods support concurrent execution. Sentence Transformers components moved to a separate integration package.
Evidence boundary: This article uses official documentation displaying version 3.3 and a local fixture running haystack-ai==3.3.0, checked on October 6, 2026. The socket exercise, document-update ledger, and diagnostic procedure are original preparation material.
Version matters when an interview provides older code. A 2.x import error can be a migration issue rather than evidence that the pipeline architecture is wrong. State the installed package and relevant integration versions, then inspect the current component signature.
Do not mechanically replace AsyncPipeline.run() with Pipeline.run() and promise the same concurrency. The migration guide explicitly distinguishes those execution paths. Also separate the framework package from a document-store integration: backend capabilities and release compatibility need their own checks.
The local exercise below uses custom components, DocumentWriter, and in-memory BM25 retrieval. It does not call a language model, download an embedding model, or test a distributed vector store. That narrow scope makes its results reproducible without claiming production guarantees.
How do you connect components without guessing sockets?
Official facts: Haystack's Creating Pipelines guide describes adding named components and connecting their inputs and outputs. The custom-component guide defines components through @component, a run() method, and declared output types.
Original socket exercise: A source accepts text: str and declares an output named text of type str. A consumer accepts text: str and returns its uppercase value. Connect the named sockets explicitly:
pipeline.connect("source.text", "consumer.text")
result = pipeline.run({"source": {"text": "socket check"}})
The saved fixture returned SOCKET CHECK. A second consumer declares a mandatory count: int input. Connecting the same string output to consumer.count raised PipelineConnectError during connection. That isolates a graph contract failure before any document retrieval or model call occurs.
Separate a nonexistent socket, an incompatible declared type, and a missing runtime input so the next diagnostic check targets the right boundary. Inspect the component's interface and pipeline.inputs() instead of assuming a component name uniquely identifies every input. Explicit sockets make a code review easier when a component has several outputs.
A declared list[Document] output also has a runtime obligation: return the expected key and valid values. A type-compatible graph can still receive empty documents or malformed metadata. Connection validation establishes interface compatibility, not the truth or usefulness of the content flowing through it.
Where should indexing stop and query execution begin?
Preparation explanation: An indexing path converts source material, cleans or splits it, attaches metadata, creates any required embeddings, and writes documents. A query path transforms the question as needed, retrieves candidates, optionally ranks them, builds model input, and generates a response.
Keep their inputs distinct. A document embedder operates on documents; a query text embedder supplies a query vector. Wiring components because both names include “embedder” is not enough. Read the output socket and the retriever's required query input.
Official fact: The DocumentWriter guide identifies it as an indexing component that writes documents to a configured store and returns documents_written. Its duplicate policy concerns documents with the same ID.
Original review question: The writer reports one document written. Does that establish that the new policy is the only policy query users can see? No. The write count does not list all retained older chunks, prove the query points at the same store, or establish that an external index has reached the application's required visibility state.
Read back the intended IDs and metadata, then execute a retrieval check. Compare the store and namespace used by indexing with those used by the serving application. A successful write into a local test store says nothing about a differently configured production retriever.
What does duplicate handling actually replace?
Official facts: DocumentWriter supports OVERWRITE, SKIP, and FAIL; NONE delegates to the document store's settings. OVERWRITE replaces a document with the same ID. Haystack's data-class documentation describes the Document ID, content, metadata, and optional embeddings.
Original update ledger: Policy version 7 has two stored chunks: policy:7:0 and policy:7:1. Version 8 has one chunk, policy:8:0. All contain the distinctive word amber so a simple query can expose retained revisions.
| Write or query operation | Observed local result |
|---|---|
| Write the two version-7 chunks | Store contains two documents |
| Overwrite policy:7:0 with revised content | Store still contains two; policy:7:1 remains |
| Write the new policy:8:0 ID | Store contains three documents |
| Retrieve amber without a version filter | Results include both versions |
| Retrieve amber with version equal to 8 | Only policy:8:0 is returned |
| Delete the two known version-7 IDs, then retrieve | Store contains and returns only version 8 |
The result follows document identity. A new version-specific ID is a new document; overwriting one old ID does not delete other IDs belonging to that source. The application must decide how a source update reconciles its complete chunk set.
Using stable positional chunk IDs creates a related problem. If a three-chunk source becomes two chunks, overwriting positions zero and one still leaves position two unless cleanup removes it. IDs that encode a content hash can also change when text changes. Define the desired source-replacement behavior before choosing an ID scheme.
How can you publish a new document revision safely?
Original design recommendation: Keep a manifest mapping a source and revision to its chunk IDs. Write the new revision, verify that its required records are available, and make the serving path select the intended revision. Retire old IDs according to the application's visibility and retention contract.
This is an application design, not an atomic transaction supplied by every Haystack document store. If the manifest and vector records live in separate systems, explain the failure window between updating them. A partial new write must not silently become a complete published revision.
Consider a crash after writing one of two required new chunks. A retry should identify the same revision and expected IDs, finish or reconcile the write, and verify completeness. Declaring the revision active solely because one writer call succeeded would expose an incomplete source.
Concurrent updates need a rule too. If version 9 completes before a slow version 8 job, the older job should not replace the serving selection with version 8. Use the source's ordering or an explicit compare-and-set contract rather than completion time alone.
Keep source deletion and access revocation in scope. Removing text from an answer cache does not remove retained retrieval records. For sensitive material, determine which serving paths can still retrieve it and what confirmation proves that the removal or filter change is effective.
What do empty or irrelevant retrieval results tell you?
Official facts: Haystack's metadata-filtering guide defines filter conditions and combinations. The InMemoryBM25Retriever guide documents lexical retrieval over its store. Its embedding retriever uses a query embedding and a configured similarity function.
Original diagnostic sequence: Begin with the eligible corpus. Count documents in the relevant store, inspect expected IDs, and check tenant, source, revision, and permission metadata. An empty result can be the correct outcome of a filter rather than a broken search algorithm.
Next inspect the query representation. For lexical retrieval, check the actual text and preprocessing. For embedding retrieval, verify that document and query vectors use compatible models, dimensions, and preprocessing. Matching vector dimensions alone does not establish that both sides occupy the same meaningful embedding space.
Then inspect candidate IDs, scores, and contents before ranking and generation. A score is specific to the retriever and configuration; it is not automatically a calibrated probability that the document answers the question. Compare results for a known-answer query and a query with no valid evidence.
Increasing top_k can recover a relevant item that was narrowly excluded, but it can also send more obsolete or irrelevant material onward. First find whether the intended item exists, is eligible, and appears among candidates. Changing the generation prompt cannot repair a document the retriever never returned.
What should a small pipeline test leave unproven?
Original test design: Start with deterministic components and documents whose expected identities are known. Check a valid connection and one invalid type connection separately. Then test a missing runtime input without involving a model provider. This keeps a graph error from being confused with a network timeout or a poor answer.
For update behavior, assert stored IDs as well as counts. A count of three does not tell you whether the store contains the two obsolete chunks and the new chunk, or three unrelated records. Query a distinctive token that exists in every test revision, then repeat with a revision filter. The returned ID set is the evidence that distinguishes retention from selection.
Include a query whose expected support is absent. Decide whether the application should return no evidence, ask for clarification, or abstain; do not require the generator to invent an answer merely because the pipeline completed. A fixture with only successful known-answer queries hides this boundary.
Finally, keep local behavior separate from deployment behavior. In-memory deletion finishing immediately does not prove an external search index has no visibility delay. A passing sequential test does not establish that two indexing workers cannot race. Use the small fixture to identify the contract, then test the selected backend and serving configuration against that contract.
How do you distinguish retrieval failure from answer failure?
Official fact: The pipeline-debugging guide describes inspecting component outputs, logging, and tracing. Intermediate output inspection can show where expected documents disappear.
Original trace review: Save the question, applied filters, retriever configuration, candidate IDs and revisions, ranking decisions, and the exact documents supplied to the prompt builder. Compare them with the cited evidence in the final answer. Avoid logging private document text unnecessarily; references and controlled samples can support diagnosis.
If version 8 appears in retrieval but a ranker drops it, evaluate the ranking stage. If the prompt builder receives only version 8 but the answer asserts an unsupported old rule, evaluate prompt content and generation. If both versions enter the prompt, revisit the revision-selection contract before asking the model to infer which is current.
Use separate evaluation outcomes for retrieval recall, selected-context relevance, supported answers, and abstention when evidence is absent. An answer that sounds fluent can still send a user to an obsolete refund rule. A correct refusal can be valuable when the eligible corpus contains no support.
The saved local run with haystack-ai 3.3.0 passed nine checks covering a valid connection, an incompatible connection, overwrite-by-ID, retained revisions, filtering, and explicit retirement. It did not test generator accuracy, embedding quality, backend transactions, or concurrent publication. Name those limits when explaining what your demonstration proves.