Articles

Network Effects A/B Testing Interview Questions: Interference and Cluster Randomization

Practice network effects A/B testing interview questions with an eight-user example, cluster randomization, spillover assumptions, and power trade-offs.

Network effects A/B testing interview questions ask what happens when one person's treatment changes another person's outcome. A messaging feature can help untreated friends. A marketplace promotion can divert demand from untreated sellers. Random assignment still matters, but an ordinary user-level comparison may answer a different question from “What would happen if we launched this for everyone?”

A strong answer identifies the interaction, defines the effect of interest, and chooses randomization and analysis around that effect. Saying “use cluster randomization” is only a starting point. You must explain cluster boundaries, remaining spillovers, precision, and the assumptions needed to interpret the result. For the broader workflow, see A/B Testing for Data Scientists.

Evidence boundary: Primary research and official engineering accounts support the attributed technical claims below. Candidate reports: no verified report establishes this exact exercise or a particular employer's interview rubric. Inference: the eight-user network, numerical calculations, design checklist, and suggested answers are original practice material. They illustrate reasoning rather than a production experiment's measured outcome.

What is interference, and which effect do you want?

Interference means a unit's outcome can depend on other units' assignments. It is different from two friends having correlated preferences even without treatment. Correlation affects uncertainty; interference changes which potential outcome is observed when neighbors receive a feature.

For an original chat-feature example, write a user's outcome as Y_i(z_i, e_i), where z_i is their own assignment and e_i is the fraction of their neighbors assigned treatment. This exposure mapping is an assumption: it says the fraction captures all relevant assignment information. If the identity of a close collaborator matters more than ten casual contacts, the mapping may be inadequate.

The rollout question might compare average outcomes under Y_i(1, 1) and Y_i(0, 0): everyone treated versus everyone untreated. A direct-effect question holds neighbor exposure fixed and changes only z_i. A spillover question holds own assignment fixed and changes neighbor exposure. Define the target before selecting the estimator.

Primary-research finding: Eckles, Karrer, and Ugander study design and analysis methods that reduce interference bias for network effects of interest, including correlated assignment and information about neighbors. This motivates an interview distinction: randomization does not automatically turn a mixed-network contrast into a full-rollout effect.

In a 50/50 user-randomized experiment, treated and control users usually interact with a mixture of assignments. Their difference can still describe a design-specific assignment effect. It need not equal the total effect of changing the whole network. Spillovers can attenuate, amplify, or even reverse the contrast relative to that target; “network effects always bias toward zero” is too strong.

Work through an eight-user network

Imagine two communities, A, B, C, D and E, F, G, H. Within each community, every user is connected to the other three. One additional edge joins D to E. There are thirteen undirected edges: six inside each community and one bridge.

Compare two illustrative assignments. First, treat A, C, E, G and leave the others in control. Second, treat the entire first community and leave the second in control. The latter is one possible cluster assignment, not a sufficiently powered experiment by itself.

Property	Mixed user assignment	Community assignment
Treated users	A, C, E, G	A, B, C, D
Edges connecting different assignments	9 of 13	1 of 13
A's treated-neighbor fraction	1/3	1
D's own assignment and neighbor fraction	Control; 3/4	Treatment; 3/4
E's own assignment and neighbor fraction	Treatment; 1/4	Control; 1/4

The community assignment aligns most interactions with treatment status, but D and E remain boundary users. Counting their entire outcomes as fully treated or fully untreated assumes away the bridge. A cut-edge count is a useful design diagnostic, not a measured causal-bias estimate: one heavily used bridge could matter more than many weak internal edges.

You can verify the edge calculation without a statistical package:

from itertools import combinations

edges = list(combinations("ABCD", 2))
edges += list(combinations("EFGH", 2))
edges += [("D", "E")]

def cut_edges(treated):
    return sum((a in treated) != (b in treated)
               for a, b in edges)

assert len(edges) == 13
assert cut_edges(set("ACEG")) == 9
assert cut_edges(set("ABCD")) == 1

Local verification: these assertions were executed for this graph. They establish the combinatorial example, not the adequacy of a clustering algorithm or the size of a real spillover effect.

How would you choose clusters in a real experiment?

Start with the mechanism. For collaboration software, shared workspaces may capture interaction. For a social feature, a pre-experiment interaction graph may be more useful than geography. For a marketplace, substitution between sellers or competition for supply can matter even when users are not friends.

Freeze a defensible pre-treatment graph and cluster definition before assignment. A graph created from treatment-period interactions may itself be an outcome of treatment. Rebuilding clusters from it can change the population and compromise the intended comparison. Track later interactions as exposure diagnostics while keeping assignment rules clear.

Evaluate several candidate partitions against cross-cluster interaction, cluster-size imbalance, baseline outcome variation, and operational feasibility. Cutting fewer edges often means larger clusters and fewer independently randomized units. The best-looking partition on a network diagram may produce a weak experiment.

Primary-research finding: Holtz and Aral's marketplace study uses simulations based on Airbnb data and finds a bias-versus-variance trade-off for graph cluster randomization. It is evidence that the trade-off can matter, not a promise of the same improvement for another marketplace.

For our chat example, we would compare workspace-based and graph-based partitions using historical collaboration activity. We would report how many users belong to several workspaces and how much communication crosses each proposed boundary. An organizational label alone does not establish an isolated interference group.

How should uncertainty and weighting change?

Analyze the design you randomized. If treatment is assigned to workspaces, treating every message as an independent assignment can substantially understate uncertainty. Depending on the estimand and assumptions, use cluster-level summaries, appropriate cluster-robust inference, or randomization-based inference that respects the assignment scheme. Few clusters require particular care; a routine large-sample standard error is not a cure.

Official engineering account: Microsoft's tenant-randomization discussion describes power challenges and why pairing tenants does not necessarily solve them. The practical implication for an interview is to ask about cluster counts, sizes, and baseline stability instead of quoting only the total number of users.

Here is a separate, simplified precision calculation. Assume equal-size independent clusters, m users per cluster, equal outcome variance, and common within-cluster correlation rho. Summing the individual variances and within-cluster covariances gives the familiar variance-inflation factor:

DE = 1 + (m - 1) * rho
m = 50, rho = 0.10
DE = 1 + 49 * 0.10 = 5.9
10,000 / 5.9 ≈ 1,695 independent-user equivalents

This original worked example shows why correlated observations carry less information. It is not a complete power analysis for an arbitrary network: unequal sizes, cross-cluster dependence, the estimator, and the number of clusters all matter. Increasing users within the same few communities does not substitute for adding well-separated randomized communities.

Weighting also defines whose effect you estimate. An unweighted average of workspace means gives a ten-person workspace the same influence as a thousand-person workspace. A user-average target requires an estimator with appropriate population and assignment weighting. State whether the decision concerns the average workspace, user, or transaction; do not let a convenient aggregation silently choose it.

Can exposure weighting recover the full-rollout effect?

Sometimes, under explicit assumptions. Primary-research finding: Ugander and colleagues' graph cluster randomization paper defines network exposure conditions and uses their assignment probabilities in inverse-probability estimators. Its unbiasedness claim depends on a correctly specified exposure model.

For a fully treated exposure, let I_i indicate that user i and all relevant neighbors receive treatment, and let pi_i be that event's probability under the actual design. A Horvitz–Thompson mean has the form sum(I_i * Y_i / pi_i) / N. Estimate the fully untreated mean analogously and take their difference for the chosen exposure contrast.

This requires positive exposure probabilities. In our two-community graph, suppose the design always assigns exactly one community to treatment. D and E can never be fully treated or fully untreated together with all their neighbors: the bridge crosses assignments in every allowed allocation. Their full-exposure probabilities are zero. No inverse weight can recover those missing exposure outcomes under that design without further assumptions or a changed target.

Independent randomization of the two communities would permit both communities to share an assignment, but two clusters remain far too little information for a useful production conclusion. Positivity is necessary, not sufficient. Tiny positive probabilities also create large weights and unstable estimates.

Randomized Graph Cluster Randomization studies randomizing the clustering itself to improve exposure probabilities and estimator behavior. Treat this as a deeper follow-up, not a reason to claim that weighting automatically removes all spillover bias. You still need a credible exposure model and a design with usable precision.

When might a switchback design be more suitable?

If users share rapidly changing supply, a stable social-graph partition may not capture interference. A region-by-time switchback assigns a whole local system to a condition for a window, then changes conditions according to a randomized schedule. The relevant question becomes how far effects travel across geographic and temporal boundaries.

Official engineering account: DoorDash's discussion of network effects, learning, and power describes these competing design goals and remaining boundary effects. In our interview reasoning, a short window may improve the number of randomized periods while increasing contamination from carryover. A long window may capture more stable behavior while leaving fewer informative comparisons.

Ask whether users need sustained exposure to learn the feature. A rapidly alternating condition may answer an immediate operational question while failing to measure a durable learning effect. Specify carryover assumptions, any transition exclusions, and time-aware inference before reading results. A switchback is a design choice with its own assumptions, not an escape from interference.

Practice defending the randomization decision

Draw the interaction first, then name the estimand, assignment unit, remaining contamination, and analysis plan.

Practice question	What to demonstrate
Choose Randomization Unit and Mitigate Network Effects	Connect the interaction mechanism to assignment
Design cluster-randomized test under network effects	Explain boundaries, cluster counts, and uncertainty
Evaluate a Live-Stream Group Notification Under Network Effects	Distinguish own exposure from friends' exposure
Design a pricing experiment with network effects	Reason about substitution and marketplace spillovers
Design and analyze a switchback experiment	Address time boundaries and carryover

For the eight-user example, a defensible answer is: “Community assignment reduces mixed-treatment edges from nine to one, but the bridge still exposes boundary users to the other condition. I would define whether we want a mixed-policy effect or a full-rollout effect, assess exposure probabilities, and power the design using clusters. I would not claim eight independent observations or assume the remaining edge is harmless.”

Start with Design cluster-randomized test under network effects, then change the bridge into a high-volume interaction. Explain why the same partition may become less credible even though its edge count has not changed.