DSPy Interview Questions: Signatures, Optimizers, and Evaluation Leakage

DSPy interview questions are rarely answered well by saying that the framework “optimizes prompts.” The useful follow-up is: what program are you optimizing, what evidence selects the winner, and what data remains untouched when you claim that it improved?

This guide builds an original support-ticket routing exercise around those boundaries. You will explain signatures and modules, choose an optimizer without confusing prompt search with weight training, and repair an evaluation that rewards the wrong behavior. The code is a documentation-grounded sketch, not a funded language-model experiment or a report of measured DSPy performance. Related practice lives in ML System Design.

The illustrated categories are conceptual. The executable-interface sketch below deliberately uses a smaller three-label taxonomy.

What does a DSPy signature specify?

A signature describes the inputs, outputs, and intended transformation of an LM-backed task. The official signature documentation presents this as a declarative interface rather than a hand-authored prompt template. It makes the task easier to compose and optimize, but does not prove that an output is correct.

For the exercise, route a ticket to billing, access, or other. The input is the customer's ticket text. The output is a category. The reference label belongs to the evaluation example, not the model's input. That distinction is essential: including a gold category in the input can make an otherwise sophisticated experiment meaningless.

from typing import Literal
import dspy

class RouteTicket(dspy.Signature):
    """Route a support ticket using the three-label policy."""
    ticket: str = dspy.InputField()
    category: Literal["billing", "access", "other"] = dspy.OutputField()

class TicketRouter(dspy.Module):
    def __init__(self):
        super().__init__()
        self.route = dspy.Predict(RouteTicket)

    def forward(self, ticket):
        return self.route(ticket=ticket)

example = dspy.Example(
    ticket="I paid twice for the same invoice.",
    category="billing",
).with_inputs("ticket")

The with_inputs("ticket") call is more consequential than it looks. It defines which fields are passed as inputs. Preserve the label for scoring, while keeping it outside the inference payload. Review other fields too: filenames, human reviewer comments, or a label-coded identifier can leak the answer even when category is excluded.

A typed output still needs runtime checks and a task metric. An adapter may enforce or parse a format, but a perfectly formatted access answer can be wrong for a billing ticket. Explain both the interface contract and the semantic correctness requirement.

How do modules differ from signatures?

A signature says what transformation is intended; a module implements a program structure that uses one or more predictors. In the sketch, TicketRouter.forward() delegates to a single Predict. A larger router might retrieve a policy, classify the ticket, and escalate uncertain cases through separate stages.

The official module documentation describes modules such as Predict and ChainOfThought and their composition. Adding a reasoning field is not evidence that reasoning is faithful or the answer is more accurate. Compare the complete program against a baseline on the same examples rather than assuming a more elaborate module must be better.

Keep intermediate fields auditable. If retrieval returns the wrong policy version, the classifier may follow it consistently and still fail the business task. If a later stage rewrites a category, inspect both the original prediction and the final result. A single aggregate score cannot localize the faulty stage.

In an interview, draw the data passed between modules and mark which fields are customer-controlled, retrieved, generated, or reference labels. This makes two kinds of bugs visible: unintended label leakage and untrusted input being treated as instructions. A signature is not an authorization boundary for tools or external actions.

Does compilation train the model's weights?

Not necessarily. DSPy's optimizer documentation distinguishes approaches that optimize instructions or demonstrations from approaches involving fine-tuning. The word compile does not by itself mean that the underlying LM's weights changed.

For ticket routing, first ask what is allowed to change. You may search instructions, select few-shot demonstrations, or adjust program components. If the proposed method trains weights, describe the training data and model deployment separately. Do not describe every optimizer as the same mechanism with different settings.

An optimizer also needs a budget. Count candidate evaluations, model calls, token consumption, wall-clock time, and any judge calls. A candidate that improves accuracy by one point while multiplying latency may be unsuitable for an interactive support workflow. Choose a budget before running search, not after seeing a favorable result.

The MIPROv2 API documentation exposes training and validation inputs for compilation. Read the installed version's signature rather than copying a remembered call. Validation used to select candidates is development data; it is not an untouched final test simply because an argument is called valset.

This article cites versioned documentation pages checked on October 4, 2026. They are not a claim that all snippets were executed against one installed DSPy release. Before running a paid experiment, pin a release, inspect the relevant API, configure an authorized LM, and run a small probe.

Repair a metric that rewards valid nonsense

Suppose the team scores the router with “output is one of the three allowed strings.” A system that always returns billing receives a perfect format score. It may route every access problem incorrectly. The optimizer has not failed to optimize; it has optimized the wrong objective.

Build a tiny metric fixture before an LM call:

allowed = {"billing", "access", "other"}
gold = ["billing", "access", "other", "access"]
predicted = ["billing"] * 4

format_rate = sum(x in allowed for x in predicted) / len(gold)
accuracy = sum(p == y for p, y in zip(predicted, gold)) / len(gold)
assert format_rate == 1.0
assert accuracy == 0.25

This deterministic example makes metric gaming visible without attributing a fabricated result to DSPy. Use validity as a diagnostic or constraint, then score task correctness against reference labels. Decide how invalid outputs, abstentions, empty responses, and provider errors affect the denominator.

If billing dominates the dataset, overall accuracy may still conceal weak access routing. Report per-class recall and a confusion matrix. If the cost of an access mistake differs from a billing mistake, document that policy and evaluate the corresponding cost; do not secretly change class weights after examining the test failures.

Human labels also need a policy. “My invoice is correct, but I cannot log in to pay it” is not unambiguously billing just because it mentions an invoice. Define routing precedence, permit an escalation category if appropriate, and adjudicate disagreements without letting the candidate's prediction determine the supposed ground truth.

Freeze the selection boundary before claiming improvement

Use training examples to build candidates, development examples to choose among them, and a final held-out test to assess the frozen choice. Freeze the taxonomy, metric, program, and acceptance rules before opening final test results. If you change them afterward, the test has become development feedback.

Split by the unit that can duplicate information. Tickets from one incident, customer conversation, or synthetic template family should not straddle splits when that would make evaluation easier than deployment. Randomly splitting rows is insufficient if each row paraphrases the same underlying problem.

Save group identifiers and assert disjointness. A useful guard checks train_groups & dev_groups, train_groups & test_groups, and dev_groups & test_groups are all empty. Also scan normalized text and near duplicates; unique IDs do not prove unique examples.

A retrieval index can leak labels too. If test tickets or their resolved answers are indexed during optimization, a clean-looking dataset split is not enough. Record the retrieval snapshot and source policy. Likewise, an LM judge that sees reference answers or candidate identity under inconsistent conditions can favor one system without measuring the intended task fairly.

The distinction from a general LLM evaluation dataset exercise is the optimization loop: DSPy may evaluate many candidates, making development reuse expected but final-test reuse especially dangerous. The interface can be elegant while the experiment remains biased.

Read a synthetic result without overstating it

Imagine a baseline gets 14 of 20 development examples right and the chosen candidate gets 18 of 20. That is a selection result on a small development set, not evidence of a 20-point improvement in future traffic. Search has deliberately favored that dataset.

On an untouched test of 40 examples, suppose the baseline gets 25 right and the candidate gets 27 right. These are illustrative counts, not measured DSPy results. The difference is two tickets. You still need paired disagreement information, the error costs, and uncertainty before deciding whether it justifies deployment.

Result record	What it supports	What it does not support
Development: 14/20 versus 18/20	Why this candidate was selected	A final generalization claim
Test: 25/40 versus 27/40	A held-out comparison on this sample	A guaranteed future uplift
Both systems valid on all examples	Output-format compliance	Correct routing
Candidate has higher mean latency	A performance trade-off to inspect	Whether tail latency meets the service target

Two systems can share the same aggregate difference while disagreeing on very different examples. Inspect which errors were fixed and introduced. A candidate that improves low-impact categories but harms urgent access cases may fail the actual decision criterion.

If the test result is disappointing, preserve it. Investigate with new development data, change the program, and obtain a new independent evaluation rather than repeatedly optimizing against the same test. The credible interview answer is a repair plan, not a renamed test set.

What should you save for reproducibility and debugging?

Save the dataset version and split hashes, input-field policy, label taxonomy, metric implementation, compiled program artifact, optimizer configuration, LM identity, generation settings, retrieval snapshot, and call budget. Record provider errors and retries instead of silently dropping difficult examples.

Seeds are useful but do not guarantee identical responses from a hosted LM, changing model alias, external retriever, or concurrent execution. Distinguish reproducible configuration from perfectly reproducible outputs. Replay a small fixed set and compare traces when a provider or framework version changes.

For a bad routing decision, inspect the sanitized input, retrieved policy, intermediate predictions, parsing result, final label, and scoring decision. Do not publish customer tickets or credentials in debugging artifacts. Store only the information necessary for diagnosis under the appropriate access controls.

Separate optimization from production permissions. A router's output may recommend a destination; it should not automatically grant an account change or issue a refund. Validate the category and apply an authorized downstream policy. Optimization can improve a task metric without making the system safe to act autonomously.

Practice the questions that expose evaluation mistakes

Question	DSPy follow-up to rehearse
Design Candidate Search And LLM Evaluation Tasks	Which data selects the candidate, and which data evaluates it?
Design an LLM quality validation system	How do format checks differ from task correctness?
Design RAG Evaluation and Debugging	Can the retrieval snapshot leak held-out answers?
Build a System to Manage Prompts and Model Behavior in Production	Which artifact and configuration can be rolled back?
Evaluate Noisy Data for LLM Post-Training	How are disputed labels handled before optimization?

Continue with ML System Design. Bring a signature, a deliberately broken metric, and a frozen-test policy to the discussion. Those three artifacts demonstrate that you understand both DSPy's programming model and the limits of an optimized score.

Sources and Further Reading

• DSPy signatures
• DSPy modules
• DSPy optimizers
• DSPy MIPROv2 API
• DSPy evaluation metrics

Documentation checked October 4, 2026. Examples and result counts are original teaching fixtures; no live LM optimization outcome is claimed.