Ratio Metric Interview Questions: User Weighting, Denominators, and Experiment Decisions

An experiment raises click-through rate from 10% to 18%. The team celebrates an 80% relative improvement. Then you discover that clicks fell from 100 to 90 while recorded impressions fell from 1,000 to 500. The fraction improved; the product outcome still needs an explanation.

Ratio metric interview questions test whether you can identify the estimand before calculating significance. Define whose outcomes count, how the denominator weights them, and which changes would justify shipping. A more precise confidence interval cannot repair the wrong question.

The worked numbers and checks below are original synthetic examples, not candidate reports or evidence about an employer's interview. Primary research and experimentation guidance are linked where used, checked September 30, 2026.

What is the difference between a pooled ratio and an average user rate?

For user i, let X_i be clicks and Y_i be impressions during the analysis window. A pooled click-through rate is sum(X_i) / sum(Y_i). An average user rate is mean(X_i / Y_i) among users for whom that fraction is defined. They are different estimators of different targets.

When every included user has positive exposure, the pooled ratio is a weighted average of individual rates, with weight Y_i / sum(Y_i). A user with 100 impressions contributes ten times the weight of a user with ten. The equal-user average gives both users the same weight.

Primary research: Deng, Knoblich, and Lu treat a ratio of aggregated quantities using the numerator, denominator, and their joint variation. Their Delta Method paper supports treating a ratio as more than an ordinary average of event-level values. Applying the Delta Method in Metric Analytics.

My recommendation is to describe the target in words first. “Clicks per impression across the observed traffic” differs from “the average exposed user's personal click rate.” Neither is universally correct. Revenue per assigned user, average order value, and conversion among exposed users each answer another product question.

State the aggregation window and assignment population too. Daily user rates averaged across days differ from one user rate over the entire week. Changing that order of operations silently changes the weights.

Can every user's rate improve while the pooled metric declines?

Yes. The original example below has two user types. Both individual click rates rise by ten percentage points, but exposure moves toward the lower-rate type.

User type	Control clicks / impressions	Treatment clicks / impressions
A	80 / 100 = 80%	9 / 10 = 90%
B	1 / 10 = 10%	20 / 100 = 20%

Control's pooled rate is 81 / 110 = 73.64%. Treatment's pooled rate is 29 / 110 = 26.36%. The pooled change is approximately −47.27 percentage points.

The equal-user averages are (80% + 10%) / 2 = 45% and (90% + 20%) / 2 = 55%. That metric improves by 10 percentage points. Both calculations are correct; they describe different weighting schemes.

This is a constructed arithmetic witness, not a causal estimate from a real two-person trial. It demonstrates why “all subgroup rates improved” does not settle the aggregate result when the weights change. It does not prove that a product's observed composition change is benign.

Explain the mechanism before invoking a paradox: type A received most control impressions, whereas type B received most treatment impressions. The denominator determines their influence. To diagnose a real experiment, inspect whether the treatment changed engagement, traffic allocation, eligibility, or recording coverage.

A standardized comparison using fixed baseline weights can answer an additional question about rates under that specified composition. Label it as a different target; do not quietly replace the prespecified primary metric after seeing which version wins.

How can denominator movement make a rate look better?

Return to the opening fixture. Control records 100 clicks and 1,000 impressions; treatment records 90 clicks and 500 impressions. The rates are 10% and 18%, so the relative rate change is (18% − 10%) / 10% = 80%.

Meanwhile, the numerator changes by −10% and the denominator by −50%. If each arm contains 100 assigned users, clicks per assigned user fall from 1.0 to 0.9, while impressions per assigned user fall from 10 to 5.

An explanation could be favorable: a cleaner interface removes low-value impressions while preserving useful actions. Another explanation could be harmful: broken rendering or telemetry eliminates opportunities and their records. The ratio alone cannot distinguish them.

Official research guidance: Microsoft's experimentation publications describe misleading metric interpretations and recommend checking data quality, affected populations, and related metrics when interpreting results. Twelve common metric interpretation pitfalls, Post-experiment analysis guidance.

My proposed diagnosis is to inspect the numerator and denominator per assigned user, relevant engagement outcomes, and instrumentation coverage. Then determine whether the denominator drop reflects an intended mechanism or a missing measurement. A statistically significant rate increase still needs that explanation.

Check units and definitions. An event-level click count can exceed impression count if repeat clicks are possible. A metric called “probability of clicking” may require a binary clicked-impression numerator instead. Renaming a count ratio as a probability does not impose the necessary bound.

What should happen to users with zero exposure?

A user with zero impressions and zero clicks has no defined personal click-through rate. In the pooled ratio, that user adds zero to both totals. If the entire arm has zero exposure, the pooled metric itself is undefined, not automatically zero.

For an equal-user rate, excluding zero-exposure users makes the target conditional on exposure. Assigning them a rate of zero creates a different metric and an explicit convention. Neither should happen invisibly in a SQL filter or null-handling expression.

Exposure can be affected by treatment. Comparing only exposed users can therefore compare different selected populations, even when assignment was randomized correctly. The original assigned-user population remains relevant for outcomes such as clicks per assigned user and probability of any exposure.

For example, suppose an interface prevents some users from opening a feed. A conditional feed click rate may improve among those who remain, while the probability of reaching the feed declines. That is a plausible mechanism to investigate, not an observed result in this article.

State which eligibility conditions existed before randomization. Distinguish those from behaviors observed afterward. Also distinguish a real zero from a missing event pipeline, an unknown identity, or an observation window that has not closed. A nullable denominator is not a diagnosis.

Which statistical unit belongs in the uncertainty calculation?

The analysis must respect the randomization and dependence structure. In a user-randomized experiment, a thousand impressions from one user are not a thousand independent randomized users. Event-level resampling can make uncertainty look artificially small when outcomes cluster within users.

Aggregate each user's paired numerator and denominator, then resample user records within each arm for a simple independent-user bootstrap. Preserve each pair. Sampling clicks independently from impressions destroys their relationship.

A compact Delta Method bridge explains that relationship. For n independent user records, let R = mean(X) / mean(Y) and define residuals Z_i = X_i − R × Y_i. An approximate variance is s_Z² / (n × mean(Y)²), where s_Z² is the sample variance of those residuals. Primary Delta Method treatment.

This approximation depends on regularity conditions, including a denominator sufficiently separated from zero. Tiny, unstable denominators deserve investigation; writing the formula does not establish that its approximation is reliable.

A user bootstrap also needs enough independent users for useful inference. Our two-type arithmetic example is deliberately too small to support a launch confidence interval. Its purpose is to reveal estimator behavior, not demonstrate statistical power.

If assignment occurs by household, store, or another cluster, the resampling unit needs to reflect that design. If units interfere with each other, neither a simple user bootstrap nor event-level independence becomes valid merely because the code runs.

What does a short Python check verify?

The original check below computes both estimators and preserves pairs during resampling. It uses only the standard library.

from random import Random

control = [(80, 100), (1, 10)]
treatment = [(9, 10), (20, 100)]

def pooled(rows):
    denominator = sum(y for x, y in rows)
    if denominator == 0:
        return None
    return sum(x for x, y in rows) / denominator

def equal_user(rows):
    rates = [x / y for x, y in rows if y > 0]
    return sum(rates) / len(rates) if rates else None

rng = Random(7)
resampled_users = [rng.choice(control)
                   for _ in control]
assert all(pair in control for pair in resampled_users)
assert equal_user(treatment) > equal_user(control)
assert pooled(treatment) < pooled(control)

The saved Python 3.12.14 check verified the direction reversal, denominator-loss arithmetic, zero-exposure conventions, and paired-record preservation. It did not estimate a trustworthy confidence interval from two records, reproduce a company experiment, or establish the coverage of a statistical procedure.

Production implementation adds stable user identity, consistent observation windows, assignment joins, and event deduplication. Explain how those produce the paired rows before discussing the library you would use for intervals.

What evidence supports an experiment decision?

A useful decision memo begins with the prespecified target and the mechanism that could change it. Report the ratio estimate and uncertainty alongside numerator and denominator outcomes on a stable assignment population.

Official research guidance: Microsoft recommends monitoring experiment integrity while the test runs, including checks that separate trustworthy measurement from treatment effects. During-experiment analysis guidance.

My proposed review has four connected questions. Did assignment and telemetry pass their integrity checks? Is the movement consistent with a plausible treatment mechanism? Are user value and operational guardrails acceptable? Does the interval support a benefit large enough to justify cost and risk?

A sample-ratio mismatch in assignment counts is different from a changed impression denominator. The first can signal assignment or analysis problems; the second may be a treatment outcome. Both need investigation, but they should not be collapsed into one label.

If the rate rises while clicks per assigned user fall, explain the trade-off rather than announcing a universal win. If the effect is uncertain, a nonsignificant result does not prove equivalence or absence of harm. Consider the interval, practical effect threshold, and additional data required.

Finally, separate exploratory metric variants from the planned decision family. Trying pooled, equal-user, exposed-only, and many subgroup definitions until one becomes significant creates another inference problem. Present those analyses as diagnostics unless the design supports their confirmatory use.

Sources and Further Reading

• Deng, Knoblich, and Lu: Applying the Delta Method in Metric Analytics
• Microsoft Research: Twelve common metric interpretation pitfalls
• Microsoft Research: Post-experiment analysis patterns
• Microsoft Research: During-experiment analysis patterns