Bayesian A/B Testing Interview Questions: Priors, Posteriors, and Decision Costs

Bayesian A/B testing interview questions usually require more than naming a prior or reporting that one variant is “likely to win.” A strong answer turns conversion counts into a posterior distribution, distinguishes a positive effect from a worthwhile effect, and connects uncertainty to the cost of shipping the wrong change.

This guide narrows the lens to Bayesian updating and decision costs.

Quick answer: For a binary conversion outcome, a Beta prior and Binomial likelihood give a Beta posterior for each arm. Compare the posterior distribution of the treatment-control difference with zero and with a business-relevant threshold. Then state the prior, experiment assumptions, decision costs, and stopping rule. There is no universal “90% means ship” cutoff.

What Bayesian A/B testing interview questions are testing

The interviewer is testing whether you can move from a statistical model to a defensible product decision. Start by identifying the decision: should the team launch, keep the control, gather more data, or run a redesigned experiment? Then confirm the metric, the randomized unit, the treatment exposure, and the period in which the outcome is counted.

Official modeling fact: PyMC’s published Bayesian A/B example models a binary conversion with a Binomial likelihood and Beta priors for each variant. It derives posterior uncertainty and relative uplift from those assumptions; it does not prescribe a universal interview answer or launch threshold. PyMC’s example also shows why prior predictive checks matter: a “weak” prior can imply a much wider range of plausible differences than its label suggests.

Build the Beta–Binomial model from the prompt

For each arm, let x be the number of users who convert out of n eligible randomized users, and let p be the underlying conversion probability. With a Beta prior, p ~ Beta(α₀, β₀), and a Binomial likelihood, x | p ~ Binomial(n, p), the posterior is:

p | x,n ~ Beta(α₀ + x, β₀ + n − x).

The conversion count adds successes and failures to the prior parameters. Beta(1,1) is uniform for one rate, but independent uniform priors still imply a broad distribution over relative lift. State the assumptions: one persistent randomized assignment per user, independent outcomes, and fixed exposure and measurement windows. Clustered or cross-exposed users may require a richer model.

Worked example: 100 versus 118 conversions

Suppose a product experiment has 2,000 eligible users in each arm. The control records 100 conversions and the variant 118. Use independent Beta(1,1) priors for this synthetic example. Those choices yield the following posteriors:

Arm	Conversions / users	Posterior	Posterior mean
Control	100 / 2,000	Beta(101, 1,901)	5.045%
Variant	118 / 2,000	Beta(119, 1,883)	5.944%

The observed difference is 0.9 percentage points, but the decision should reflect uncertainty around the underlying rates. Draw from each posterior and subtract the control rate from the variant rate. A reproducible NumPy simulation looks like this:

import numpy as np

rng = np.random.default_rng(42)
p_control = rng.beta(101, 1901, size=1_000_000)
p_variant = rng.beta(119, 1883, size=1_000_000)
delta = p_variant - p_control

print(np.mean(delta > 0))          # about 0.894
print(np.mean(delta > 0.005))      # about 0.710
print(np.quantile(delta, [0.025, 0.975]))

The simulated posterior places about 89.4% probability on a positive difference and about 71.0% on a lift above 0.5 percentage points. Its 95% credible interval for the absolute difference is roughly −0.51 to +2.31 percentage points. Those are approximate Monte Carlo summaries for this prior and toy dataset, not company results. The interval still includes a small negative effect even though the probability of any positive effect is high.

“Probably better” is not the same as “worth shipping”

Posterior questions need a threshold that matches the decision. P(Δ > 0) asks how likely the variant is better than control. P(Δ > 0.005) asks how likely it clears a 0.5-point minimum effect. A probability such as “89% chance of beating control” does not tell a product team whether 0.1 points is valuable enough to cover engineering work, support costs, or user risk.

Name the scale: a 0.5-point increase from 5% to 5.5% is a 10% relative lift. Set the minimum worthwhile effect from the decision context, and ask what benefit or harm would justify rollout.

If the decision is whether the effect is exactly zero, note a model limitation: continuous priors assign probability zero to one exact point such as Δ = 0. They can estimate the probability of a practically negligible range, or ROPE, but evidence for a point null requires an explicit model that includes that possibility. The Journal of Statistical Software article Informed Bayesian Inference for the A/B Test discusses this distinction between estimating rates and comparing a model that allows no difference.

Put implementation cost and error cost into the decision

Translate the posterior into a quantity the stakeholder can act on. Suppose the next rollout would reach 1,000,000 eligible users, each incremental conversion is worth $8 in contribution margin, and shipping costs $10,000. Under these illustrative assumptions, the break-even absolute lift is:

$10,000 / (1,000,000 × $8) = 0.00125 = 0.125 percentage points.

The posterior mean difference in the example is about 0.899 percentage points. Multiplying that mean by future traffic and margin implies approximately $71,928 in expected incremental margin before rollout cost, or about $61,928 net. The posterior probability of exceeding the 0.125-point break-even threshold is about 85.9%. Under a risk-neutral expected-net-value rule and these assumptions, shipping beats keeping control; a binding harm guardrail or greater risk aversion could reverse that choice. Both quantities depend on the assumed future traffic, margin, cost, model, and prior; neither is a forecast that the company will realize that return.

Decision quantity	Example result	What it answers
Probability of any lift	89.4%	How likely is Δ above zero?
Probability above practical threshold	71.0% above 0.5 pp	How likely is the effect large enough to matter?
Probability above break-even	85.9% above 0.125 pp	How likely is expected benefit to exceed the stated rollout cost?

A decision-theoretic framing compares actions by their expected utility or expected loss. Inference: In an interview, this is often a clearer close than selecting a universal posterior cutoff, because launch cost and downside vary by product. A false positive can expose users to harm; a false negative can delay a valuable change; more runtime has an opportunity cost. Goldberg and Johndrow’s decision-theoretic A/B testing paper formalizes an adoption decision through an action set, loss function, and Bayes risk. The inputs still come from the decision-maker: a precise calculation cannot rescue implausible costs.

Choose a prior you can explain

A prior is a model input, not a decoration added after seeing results. Beta(1,1) keeps an interview calculation simple. A historical prior requires a reason that the earlier traffic and metric are comparable, plus a sensitivity check for mismatch.

Show sensitivity under at least one reasonable alternative. For the same synthetic counts, a prior of Beta(2,38) has a 5% mean and an effective prior sample size of 40 per arm. It produces a posterior probability of a positive difference of about 89.4%, and a probability of exceeding 0.5 points of about 70.5%—close to the Beta(1,1) results in this example. That stability is specific to these counts and priors. With fewer observations, a stronger or poorly matched prior can move the answer substantially.

Defend a prior by naming its source, checking predictions before updating, and showing how reasonable alternatives change the decision. Reusing the test data as historical prior evidence double-counts it. Shared baselines or many variants may need a joint or hierarchical model.

Bayesian updating does not erase experiment planning

The posterior can be updated as data arrive, but “Bayesian” is not a blanket permission to stop any test whenever a dashboard crosses a chosen number. The Microsoft Research paper Continuous Monitoring of A/B Tests without Pain establishes results for proper Bayesian stopping rules and discusses improper practices. It does not say that every repeatedly checked threshold controls false launch risk or expected loss.

Before the test, set the population, metric, analysis unit, minimum runtime or information target, guardrails, and stopping rule. Simulate plausible effects and priors to estimate how often the rule makes a good or bad decision. Sample-size and duration planning still matter: the experiment uses traffic and time, and must resolve effects that could change the action.

Check the assumptions before trusting the posterior

The basic example assumes a stable binary outcome, valid randomization, independent users, and consistent exposure measurement. In practice, inspect assignment integrity and sample-ratio mismatch; check that treatment does not change who enters the analysis, and confirm each user is counted once. If users influence one another, a user-level independent model misses interference; if randomization is by market or team, account for clustering.

Use a model that fits the metric. A Beta–Binomial setup is for binary outcomes such as converted or did not convert. Revenue per visitor, average order value, rates with changing denominators, repeated purchases, and long-tailed values need models that represent their sampling and dependence structures. Guardrail metrics also matter: a positive conversion posterior does not justify rollout if latency, refunds, or user complaints cross an agreed harm limit.

Keep the estimand stable. A randomized experiment estimates an effect for its defined population and exposure; switching to a post-treatment subgroup or incomplete conversion window changes the question. Bayesian computation describes uncertainty conditional on the model; it does not verify assignment or population fit.

Candidate-reported practice prompts

Question	Bayesian angle to practice
Design an A/B test for search ranking	Define the user-level estimand, primary metric, and uncertainty summary before discussing launch.
Design a free-month experiment	Explain how delayed revenue and asymmetric downside affect the value of a conversion lift.
Design an A/B test with causal inference	State assignment assumptions and distinguish model uncertainty from broken randomization.
Analyze A/B Test Results to Inform Stakeholder Decisions	Translate a posterior interval and practical threshold into a stakeholder-ready recommendation.
Design and analyze pricing-page A/B test	Compare posterior benefit with rollout cost, revenue guardrails, and a predeclared stopping rule.

Inference: These practice prompts make it sensible to rehearse decision-focused experiment reasoning. They do not show how often Bayesian methods appear in interviews or verify the live hiring loops at the companies named. If an interviewer does not specify an inference framework, ask whether they want a Bayesian or frequentist analysis—or state your chosen model and make the comparison explicit.

A concise interview answer

A clean response can fit into a minute: “I’ll define one conversion per randomized user within the stated window and confirm exposure and assignment integrity. For a binary outcome, I can use a Binomial likelihood with a prior that I’ll justify and sensitivity-check, then report the posterior distribution of the absolute lift. I’ll distinguish the probability of any improvement from the probability of clearing a business threshold. Finally, I’ll compare posterior expected value with rollout cost and guardrails, and use a stopping rule that was set before looking at interim results.”

Then work one numerical example, show what a different prior changes, and ask what the team considers a worthwhile effect. That answer makes the probability, threshold, costs, and assumptions visible without implying that Bayesian inference removes the need for a careful experiment.

Choose one prompt, define the estimand, and explain how the decision changes when you vary the practical threshold or rollout cost.

Sources and Further Reading

• PyMC: Introduction to Bayesian A/B Testing — Binomial conversion models, Beta priors, uplift, and prior-predictive checks.
• Deng, Lu, and Chen: Continuous Monitoring of A/B Tests without Pain — Bayesian testing and conditions on stopping rules.
• Gronau et al.: Informed Bayesian Inference for the A/B Test — Priors, posterior estimation, and the point-null distinction.
• Goldberg and Johndrow: A Decision Theoretic Approach to A/B Testing — Expected loss and feature-adoption decisions.

Research checked September 12, 2026. Numerical results are from the stated synthetic model; interview records are not guarantees of future questions.