Experimentation Platform System Design Interview: Assignment, Exposure Logs, and Metric Integrity

An experimentation platform system design interview can be ruined by perfect statistics on corrupted data. If assignment changes between devices, exposure is logged for users who never saw the feature, or one variation loses events, a significant result may be an instrumentation artifact.

LaunchDarkly officially documents that evaluation events determine experiment exposure and provides raw-exposure validation and sample-ratio mismatch checks. Those product behaviors illustrate the contract. The storage layout, identity resolution, and ramp policy below are editorial inference. This is a system-design guide, not an A/B statistics course.

For a closely related drill, use Design A/B testing platform.

Experimentation Platform Interview: Quick Answer

Create an immutable experiment version with eligibility, randomization unit, allocation, variation weights, mutual exclusions, start time, and metric definitions. Assignment is a deterministic hash of experiment salt and stable unit ID, stored or recomputable. Log exposure only when the application evaluates and actually uses the experimental value under the declared exposure rule.

Ingest assignments, exposures, and metrics into versioned schemas. Join by canonical randomization unit and bounded attribution window. Before estimating effects, run sample-ratio, duplicate, missing-ID, cross-variation, timestamp, and late-data checks. A failed integrity gate pauses interpretation and possibly ramping.

The Core Contract

An experimentation platform is a causal data contract: deterministic assignment, actual exposure, and metric identity must join without selective loss before statistical analysis is meaningful.

Event	Required identity	Key time	Integrity question
assignment	experiment version + unit	decision time	is the bucket stable?
exposure	assignment + surface	first/each use	did the unit actually receive treatment?
metric	subject + metric schema	outcome time	is it in the attribution window?
exclusion	namespace + unit	enrollment time	can the unit enter another test?
ramp change	experiment version + allocation	activation time	was analysis segmented by allocation era?

The platform should preserve these separate events. Inferring exposure from assignment alone changes the estimand.

Make Assignment Deterministic and Versioned

Select a randomization unit that matches interference and product behavior: user, account, device, session, store, or region. Hash a namespaced experiment salt with the canonical ID into a fixed bucket range. Weight intervals map buckets to control or treatment. Stable inputs produce stable assignment without a central write on every request.

Changing weights can preserve existing treatment membership or rebucket units; state the policy. Freeze material definition changes into a new experiment version. Anonymous-to-logged-in identity transitions need a rule, because merging them after exposure can move outcomes across variations.

Log Exposure at the Product Decision

Assignment says what a unit would receive. Exposure says the product evaluated and used the variation in a context that could affect the metric. Emit experiment/version, variation, unit, evaluation reason, feature version, request ID, and event time. Deduplicate repeated exposures according to the analysis contract while retaining raw events.

Logging only in a downstream analytics callback may selectively lose slow or failed treatment requests. Logging before the value is used may count invisible users. Put the event next to the decision and monitor variation-specific delivery, latency, and missing fields.

Protect Metric Identity and Join Semantics

Metric definitions include numerator, denominator, inclusion filters, aggregation unit, attribution window, and late-data policy. Version them before the test starts. Join outcomes to exposure by canonical unit and time; do not select a convenient post-treatment subset. Revenue and count metrics require deduplicated source events, not page-refresh totals.

Keep raw events and reproducible transformations. Backfills create a new dataset generation with code version and input ranges. Analysts can compare generations before promotion. Privacy deletion and retention must propagate through derived datasets without silently changing only one variation.

Gate Decisions on Integrity Before Effect Size

Check observed assignment counts against configured allocation, exposure/assignment conversion by variation, duplicate and cross-variation units, unknown experiment versions, event lag, and metric availability. Sample-ratio mismatch is an alarm, not a diagnosis; causes include hashing bugs, eligibility drift, event loss, bot filters, and identity merges.

Ramp through predetermined cohorts and guardrails. Stop for integrity failure, severe product regression, or safety risk. Do not stop merely because an early noisy estimate looks attractive. Record every allocation change and analysis cutoff so repeated peeking and operational decisions remain reviewable.

Work Through a Failure Trace

For experiment v3, hash 100,000 account IDs into 50/50 allocation. The fixture repeats assignment across processes, emits exposures for only rendered experiences, duplicates 1% of metric events, and drops 5% of treatment exposures. Deduplication fixes metric counts, while the exposure-rate asymmetry and sample-ratio checks remain visible and block interpretation.

Experimentation Platform check: What is the randomization unit and how is assignment kept stable? Walk the trace once in event-time order, once with retries and reordering, and once after rebuilding projections from durable records. State the expected invariant after every step. If the outcome depends on message arrival order when the contract says it should not, the design is missing an identity, version, or transaction boundary. If the design intentionally permits a temporary difference, give it a watermark, maximum age, and repair owner.

Experimentation Platform check: When does assignment become an exposure? This trace is an interview model, not a measured production benchmark. Add a skew test concentrated on one tenant or key, a two-times-peak capacity test, and a fault injection at the acknowledgment boundary. Preserve input versions and random seeds so a failed run can be reduced and replayed.

Turn the Requirements into Verification

Write these five questions beside the diagram: What is the randomization unit and how is assignment kept stable? When does assignment become an exposure? How are exposure and outcome events joined without bias? How are overlapping experiments and exclusions handled? Which integrity checks stop a ramp? Each question must point to a record, transition, test, and observable signal. If one can be answered only with 'the service handles it,' the contract is still incomplete.

Experimentation Platform check: How are exposure and outcome events joined without bias? Build a deterministic fixture with five to ten timestamped inputs. Include an identical retry, a conflicting reuse of an ID, a late or stale input, a worker failure after durable work but before acknowledgment, and a rebuild of the serving projection. Expected results must be written independently of the implementation. Verify the authoritative state, the serving state through a named watermark, and every emitted correction or rejection reason.

Experimentation Platform check: How are overlapping experiments and exclusions handled? Then add property checks. Applied logical IDs remain unique; versions never move backward; reserved or consumed value obeys the topic's invariant; a rebuild through the same input boundary produces the same digest; and an unauthorized reader cannot cross its scope. Generate reordered deliveries around the boundary instead of testing only the happy sequence. A passing fixture proves these declared cases, not production throughput or every network failure.

Experimentation Platform check: Which integrity checks stop a ramp? For fault testing, stop a worker immediately before and after its commit, duplicate queue delivery, delay one partition, and make the read projection unavailable. The runbook should say whether intake pauses, work accumulates durably, readers serve a last-known-good generation, or the request fails closed. Record the recovery point and compare the rebuilt digest before switching traffic. This is concrete evidence that the recovery story matches the write contract.

Capacity, Operations, and Recovery

Experimentation Platform check: What is the randomization unit and how is assignment kept stable? Estimate the durable write rate, average and peak payload, partition skew, retention volume, read or query shape, and acceptable lag. Convert those assumptions into partitions, worker concurrency, storage tiers, and backpressure. Numbers are sizing inputs, not universal facts. Say which limit fails first and what the user sees when it does.

Experimentation Platform check: When does assignment become an exposure? Operate from contract metrics: accepted-versus-applied counts, duplicate/conflict rate, oldest pending work, authoritative-versus-serving divergence, retry exhaustion, and p95/p99 latency. Alert on sustained error-budget consumption and correctness gaps rather than raw traffic. Keep a runbook that can pause intake or rollout, drain work, rebuild a named projection through watermark W, verify a checksum, and switch readers to the new generation.

Experimentation Platform check: How are exposure and outcome events joined without bias? Use least privilege between producers, workers, control plane, data plane, and investigators. Encrypt transit and storage, rotate credentials, audit high-risk actions, and avoid placing secrets or unnecessary personal data in operational logs. Security controls support the data contract; they do not replace idempotency or verification.

Misleading Answers and Better Follow-Ups

Experimentation Platform check: How are overlapping experiments and exclusions handled? A weak answer names Kafka, Redis, and a database before defining truth. Another says “exactly once” without identifying the idempotency key, persistence boundary, or replay behavior. A third adds a cache but cannot explain invalidation or rebuild. Replace each slogan with one invariant, one linearization point, and one counterexample.

Experimentation Platform check: Which integrity checks stop a ramp? Ask the interviewer to change a requirement: permit temporary staleness, add multiple regions, require a correction after finalization, or remove a dependency. Explain which state transition, key, and monitor changes. That is more useful than redrawing the same boxes at a larger scale.

Experimentation Platform check: What is the randomization unit and how is assignment kept stable? Separate three evidence classes in the answer. Official documentation supports the cited product or protocol behavior. The proposed architecture, thresholds, and sequencing are inference from the stated requirements. This prevents a sensible design choice from being presented as an external fact.

A 45-Minute Interview Plan

Spend the first five minutes clarifying actors, scale, correctness, retention, and failure expectations. Use the next eight to define the authoritative records, IDs, versions, and invariants. Draw the write path and its acknowledgment boundary for ten minutes, then the read or serving path for five.

Use ten minutes for the title-specific failure trace, including one retry, one stale or late event, and one recovery. Reserve the final seven minutes for sizing, skew, access control, monitors, and trade-offs. Keep a short list of assumptions visible. When the interviewer changes one, update the affected contract before changing components.

End with the source of truth, the permitted temporary inconsistency, and the repair procedure. A compact design that can be replayed and verified is stronger than a large architecture with undefined states.

Final Takeaway

For a experimentation platform system design interview, lead with this thesis: An experimentation platform is a causal data contract: deterministic assignment, actual exposure, and metric identity must join without selective loss before statistical analysis is meaningful. Define the evidence boundary, run a concrete race or failure trace, and show how the system detects and repairs divergence.

Sources and Further Reading

• LaunchDarkly experimentation events
• LaunchDarkly exposure validation
• LaunchDarkly experiment configuration

Official sources support only the cited mechanism. Architecture recommendations, capacity figures, and interview sequencing are editorial inference.