Experiment Noncompliance Interview Questions: Intent-to-Treat and Treatment-on-the-Treated

A product experiment randomly assigns users to a new onboarding flow, but only some assigned users actually see it. The launch dashboard says conversion rose two percentage points. An interviewer asks: “What is the effect on the people who saw the treatment?” Dividing conversions among viewers by conversions among nonviewers feels natural—and can destroy the benefit of randomization.

The first move is to separate assignment, receipt, and outcome. Then name the estimand before computing it. This guide uses an original numerical exercise to show what intent-to-treat (ITT) identifies, why an exposed-only comparison can be biased, and when a complier or treatment-on-the-treated interpretation needs extra assumptions. It is statistical preparation, not evidence that a specific company asks this prompt. For adjacent practice, Design an A/B test with guardrails and SRM checks addresses experiment integrity before the effect estimate.

Start by drawing the three clocks

Imagine 2,000 eligible users randomized equally. Z = 1 means assignment to the new flow; Z = 0 means assignment to control. D = 1 means the new flow was actually received, using a prespecified exposure definition. Y = 1 means conversion during the same follow-up window. Assignment occurs first; exposure follows; conversion is measured later. If the logs put conversion before first exposure, do not quietly treat that user as exposed for a causal analysis.

Noncompliance can mean an assigned-treatment user never sees the new flow, perhaps because the relevant page was not reached or a client version is old. A control user might receive it through a configuration leak. Those are different operational failures. Missing exposure telemetry is different again: a missing log does not prove nonreceipt. Begin the interview by asking which mechanism the data represent.

The randomized comparison is between all users assigned treatment and all users assigned control, whether or not each user saw the feature. That is the ITT effect of offering or assigning the new flow under the experiment's actual delivery conditions. It is often the policy-relevant launch quantity: if the rollout mechanism is unchanged, what happens when we assign this experience? Methodological work on imperfect compliance distinguishes that assignment effect from effects of receiving treatment; see the trial-estimand analysis.

This distinction prevents a common verbal error. A small ITT does not prove the feature has little effect when used. Low uptake can dilute an assignment effect. Conversely, a large exposed-only contrast does not prove the feature works; people who reach the feature may differ in intent, activity, device, or eligibility.

Work a numeric interview case

Here is a deliberately simple one-sided noncompliance case. No control user receives the new flow. Both arms have complete conversion outcomes and the same follow-up definition.

Assignment	Users	Received new flow	Converted	Conversion by assignment
New-flow assignment	1,000	600	120	12%
Control assignment	1,000	0	100	10%

The ITT estimate is 12% − 10% = 2 percentage points. That calculation keeps the randomization intact. State both the absolute effect and the unit; “2% uplift” could mean a relative lift and is ambiguous. The relative increase in this toy example would be 20% compared with the 10% control rate, a different number.

Suppose the 600 treatment viewers account for 90 of the 120 treatment-arm conversions. Their observed conversion rate is 15%. The 400 assigned-treatment nonviewers account for the other 30, a 7.5% rate. Comparing 15% for viewers with 10% for everyone assigned control yields a five-percentage-point contrast. It is not automatically a causal “effect on the treated.” Viewers were selected after randomization. Perhaps they reached onboarding because they were already more motivated.

The 7.5% rate among nonviewers also does not establish that the feature hurts anyone. It is another selected subgroup. Randomization balances groups by Z in expectation, not by the postassignment D variable. Conditioning on exposure can open selection paths and distort the comparison. The answer to “why not compare exposed with unexposed?” is not “because the sample is smaller”; it is “because receipt is no longer randomized.”

When can assignment identify a receipt effect?

If assignment changes exposure and meets instrumental-variable assumptions, the ratio of the ITT effect on Y to the ITT effect on D estimates a complier average causal effect (CACE), also called a local average treatment effect in many settings. In this one-sided example, the exposure difference is 60% − 0% = 60 percentage points. The illustrative Wald ratio is 2 pp / 0.60 = 3.33 percentage points.

Do not present 3.33 pp as a free correction factor. It depends on substantive assumptions. Assignment must be randomized as claimed; it must affect the probability of receipt (a nonzero first stage); it must affect conversion only through the defined receipt, rather than through some other message or expectation; and monotonicity must rule out units who would receive treatment only under control assignment. No interference between users is also relevant to the simple potential-outcomes story. These restrictions are not all testable from the observed table. The noncompliance methods literature treats assumptions and missing outcomes as central, not footnotes.

In the toy numbers, the exposed users convert at 15%, but their unobserved counterfactual conversion under control is not the full control group's 10%. Under the simplifying model, 40% are never-takers with a 7.5% conversion rate in both assignments. The control group's 10% can then be decomposed as 0.6 × 11.67% + 0.4 × 7.5%. The implied control outcome for compliers is about 11.67%, so 15% − 11.67% ≈ 3.33 pp. This is a consistency demonstration under assumptions, not a way to label individual control users as compliers from the table alone.

The phrase “treatment-on-the-treated” is used inconsistently across teams. Some interviewers mean an instrumental-variable effect among compliers; some mean a descriptive outcome among users who received treatment; others mean an average treatment effect for the actually treated (ATT). Ask which estimand they want. The Wald ratio targets compliers under its assumptions, not an arbitrary exposed-user ATT in every design. If compliance is two-sided, the first stage is P(D=1|Z=1) − P(D=1|Z=0), not merely the treatment-arm exposure rate.

The exclusion restriction deserves a concrete challenge. Imagine the treatment assignment sends an email announcing a new flow. Some recipients never open the flow, but the email itself reminds them to subscribe. Assignment may now change conversion without changing defined receipt, so dividing the ITT by the receipt first stage attributes the email's direct effect to the flow. Similarly, a tiny first stage makes the ratio unstable: a small denominator amplifies outcome noise and often produces a wide interval. A careful candidate would report the first-stage estimate and uncertainty, investigate the assignment experience, and avoid a precise-looking point estimate unsupported by the design. Neither the email mechanism nor the first-stage problem is repaired by simply dropping nonviewers.

Map operational failures to statistical consequences

Before interpreting a weak ITT, inspect the delivery system. Check assignment counts and sample-ratio mismatch, exposure logging completeness, client version, targeting rules, and whether the control experience leaked into treatment or vice versa. Microsoft's experimentation guidance treats data quality as foundational to trustworthy analysis; a broken assignment or missing event pipeline is not “interesting noncompliance” until measurement is reconciled.

Then ask whether the outcome window begins at assignment or exposure. Starting a conversion window at each user's first exposure excludes assigned users who never expose and can create immortal-time or selection problems. A policy-effect analysis typically anchors follow-up at assignment or another common pre-treatment event. If the question is specifically about postexposure behavior, state how the cohort and clock change and what causal claim you give up.

Missing Y is not the same as Y = 0. If users disappear from outcome tracking, a clean ITT rate cannot be computed by silently dropping them. Audit why outcomes are missing, whether missingness differs by assignment, and whether bounds or sensitivity analyses are needed. An exposure log missing at random is not guaranteed either; app crashes and older clients can be related to both receipt and outcomes.

For a switchback or cluster experiment, the assignment unit may be a time block or market rather than a user. Do not reuse individual-user standard errors when treatment can spill across units. The estimand, variance calculation, and possible noncompliance mechanism follow the randomization design.

The interview decision table

Use the following original table to keep the answer disciplined:

Question from the interviewer	First quantity to report	What would justify more
“Did assigning the feature improve conversion?”	ITT by randomized assignment	Validate assignment, follow-up, and outcome capture
“Why is the observed effect small?”	ITT plus receipt rates by assignment	Investigate delivery and telemetry before blaming the feature
“What is the effect among people who would comply?”	CACE/IV candidate, clearly labeled	Randomization, first stage, exclusion, monotonicity, sensitivity
“What did actual viewers do?”	Descriptive viewer rate	Do not call the selected contrast causal
“Should we launch?”	Assignment-policy benefit and guardrails	Consider rollout reach, future uptake, uncertainty, and harms

This table also prevents the opposite overreach: refusing to discuss receipt at all. Exposure is essential for diagnosing implementation and for asking a different causal question. It simply does not inherit randomization automatically.

A good spoken answer to the toy prompt

“We randomized 1,000 users per arm. The assignment-based conversion rates are 12% and 10%, so the ITT is +2 percentage points. Only 60% of assigned-treatment users received the new flow, so that ITT is the effect of assignment under the observed delivery mechanism, not necessarily the effect of receipt. The 15% conversion rate among viewers is descriptive and selected. If assignment is a valid instrument, the first stage is 0.60 and the Wald estimate for compliers is about +3.33 percentage points, but I would explicitly check exclusion, monotonicity, missing outcomes, and the exposure definition before presenting it as causal. For a launch decision, I would report the ITT, uptake, and guardrails, then model how a changed rollout might alter uptake rather than scaling the effect mechanically.”

That answer names the quantity first, performs the arithmetic second, and lists the limits third. It resists both the exposed-only shortcut and the temptation to present an IV estimate without its assumptions.

Practice questions to connect design and analysis

Question	What to practice
Design an A/B test with guardrails and SRM checks	Validate randomization before effect interpretation
Debug and harden trial-assignment Python code	Trace implementation failures in assignment
Design an email flu-shot experiment	Distinguish delivery from behavioral response
Analyze A/B Test Results for Subscription Conversion Rates	Define denominator and conversion window
Recommend a Decision After a Nonsignificant Experiment	Translate uncertainty into a decision

Sources and further reading

• Handling Missing Data in Randomized Experiments with Noncompliance — methodological treatment of noncompliance and missing outcomes.
• Sensitivity of estimands in clinical trials with imperfect compliance — distinction between assignment and complier estimands.
• Microsoft Research: Data quality as a foundation for trustworthy A/B testing — experiment-quality context.