Distributed Counter Interview: Accuracy, Hot Keys, and Reconciliation

A distributed counter interview is rarely about whether you know INCR. The hard part is defining what the number means when clients retry, one key attracts most writes, replicas lag, and serving state must be rebuilt. Start by asking whether the product needs an exact financial total, a monotonic display, or a bounded-staleness estimate. Those contracts lead to different write paths.

Redis officially documents INCR as an atomic operation for one key. It also documents cluster hash slots, which imply that a single key belongs to one slot even when the cluster has many nodes. Those are useful primitives, but they do not make a hot global counter horizontally scalable or make a multi-region total exactly once. The design must add identity, partitioning, and reconciliation.

Distributed Counter Interview: Quick Answer

Define three states. The authoritative state is a durable ledger or database total with an idempotency rule. The aggregation state groups accepted deltas by counter, time bucket, or shard. The serving state is optimized for fast reads and may be seconds behind. Every accepted logical update has an event_id, counter_id, signed delta, and event time; a duplicate event changes the total zero additional times.

For a hot key, shard the logical counter into physical stripes such as views:video42:0..63, selected by a stable writer or request hash. Reads sum stripes or use an asynchronously materialized total. State the consequence: exact read-on-write across every stripe costs fanout or coordination, while cached totals trade freshness for throughput. Reconciliation recomputes an expected total from the authoritative state and applies or rebuilds the serving difference.

Clarify Accuracy Before Drawing Components

Ask which errors are unacceptable. A billing counter may require every accepted charge exactly once and allow a slower query. A view counter may accept a displayed total that trails by five seconds but should never decrease. An inventory counter may need conditional updates because the invariant is available >= 0, not merely addition.

Separate duplicate error, loss error, and staleness. At-least-once delivery without deduplication creates positive error; acknowledging before durable storage creates negative error after a crash; a correct queued event creates temporary staleness. Give each a metric. “Within 0.1%” is incomplete unless you say whether that bound is per read, per minute, or after reconciliation. This contract decides whether you can batch, sample, or serve a cached aggregate.

Design an Idempotent Write Path

Accept POST /counters/{id}/events with an idempotency key generated for the logical action. Validate authorization and bounds, persist the event or transactional total, then acknowledge. A consumer may replay the event, so it must record the event ID with the applied mutation or derive a versioned total that makes replay harmless.

A common wrong answer puts a distributed lock around every increment. The lock serializes traffic and still does not define what happens if the client times out after commit. Idempotency addresses that ambiguity directly. If storing every event ID forever is too expensive, scope the dedupe window to the producer's retry horizon, use a compact ledger keyed by (counter_id, event_id), and explain the residual risk after expiry. For strict financial history, retain the durable event instead of expiring proof.

Break a Hot Logical Key into Physical Stripes

Ordinary sharding by counter_id cannot spread one celebrity counter because every update hashes to the same partition. Add a stripe dimension. With 64 stripes and writers distributed uniformly, the hottest logical key can use 64 independent atomic updates. A writer can choose a stripe from its stable ID to reduce churn; random selection is simpler but makes retry identity essential.

The read path now has choices. Fan out to all 64 stripes for an exact current sum, maintain a cached aggregate through a stream, or combine a recent cached total with newer bucket deltas. Fanout gives freshness but multiplies read work. Materialization makes reads cheap but introduces lag. Adaptive striping limits overhead: ordinary counters use one stripe and hot counters move to a larger generation recorded in metadata. Never silently change generations; readers must know which stripes remain part of the total.

Reconcile from a Named Source of Truth

Suppose the ledger contains accepted deltas totaling 1,004,230, while the serving stripes sum to 1,004,227. The signed difference is +3. Do not blindly issue INCRBY 3 while consumers are still applying events; that can double-count the late records. Capture a ledger watermark W, compare it with a serving snapshot applied through W, and repair to a versioned expected total. Then resume after the same watermark.

Prefer replacement or generation rebuild over an unversioned corrective delta when ownership is unclear. Write a new aggregate generation, verify its checksum and event range, then atomically switch the read pointer. Keep the old generation until rollback expires. Reconciliation metrics should include ledger-versus-serving difference, oldest unapplied event age, duplicate rejection count, stripe skew, and repair duration. An alert on the raw total alone cannot distinguish growth from corruption.

Handle Failures Without Inventing Exactly Once

A producer timeout after durable commit creates an ambiguous retry; the idempotency key resolves it. A consumer crash after applying but before acknowledging causes replay; the applied-event record resolves it. A lost cache shard is recovered from the ledger or snapshot plus later events. A regional partition may allow local increments that merge later, but then the answer must define whether negative deltas and invariant checks commute.

CRDT counters can converge for suitable operations, but “eventual convergence” does not preserve every product invariant. A grow-only counter cannot decrement; a positive-negative counter can merge signed components but does not prevent overselling. If the interviewer requires global non-negative inventory, choose coordinated ownership or reservations. This distinction shows judgment: availability-friendly arithmetic is useful only when the business rule is also mergeable.

Use a Numerical Capacity and Freshness Trace

Assume one viral counter receives 320,000 updates per second, reads arrive at 20,000 per second, and the product accepts five-second staleness. With 64 stripes, the average write rate is 5,000 per stripe per second before skew. Aggregate each stripe every second and publish a version containing all stripe watermarks. The serving API reads one materialized value instead of 64 keys.

This is a sizing example, not a measured benchmark. Ask what happens at two times the estimate. Backpressure the aggregation consumer without dropping durable events, expose lag, and let the displayed number fall behind within the declared budget. If the five-second budget is exceeded, the API can label the value delayed or fall back to a slower authoritative query for privileged use. The interview win is connecting capacity to visible semantics rather than reciting node counts.

Compare Counter Designs Against the Contract

An exact single-owner counter is the cleanest answer when one partition can handle the peak: the owner serializes updates, stores a durable sequence, and serves read-after-write state. Its limit is availability and throughput at that owner. A striped counter spreads writes and preserves exact addition once all stripes are included, but an exact fresh read now requires fanout or a synchronized materialization watermark. A probabilistic sketch gives excellent memory and merge properties for distinct-count questions, but it does not answer an exact balance request. Name the data structure only after the query is fixed.

Also distinguish operation accuracy from read accuracy. The system can accept every event exactly once while a cached read is temporarily stale. Conversely, a read can look fresh while duplicates have permanently inflated the authoritative total. Put these dimensions in the API: return an as_of watermark or version with the value, and document whether clients may require min_version for read-your-write behavior. For a UI view count, it may be acceptable to round and update periodically. For quota enforcement, use a conditional authoritative write rather than trusting the rounded display.

Discuss decrements separately. If updates can be negative, validate who may issue them and whether the invariant allows the counter below zero. A compensation event should reference the original event rather than silently deleting history. During a rebuild, sort or deduplicate by stable event identity, not arrival time. These choices make audit and reconciliation explainable even when events arrive late or out of order.

Make the Answer Reviewable

Turn the distributed counter design into a small deterministic review before adding more scale. Write the initial state, five to ten timestamped inputs, every expected state transition, and the exact read result at two watermarks. Replay one input, deliver another out of order, pause one consumer, and rebuild the serving view from its durable boundary. The replay should not create an extra effect, the stale input should follow the declared version rule, and the rebuilt read should match the authoritative result through the same watermark.

Keep expected results independent of the implementation. If the same code produces both the output and the assertion, a shared assumption can hide the defect. Add one invariant check that scans all state, even if production serves a faster index. For a distributed design, say which part this fixture does not prove: it does not measure production latency, network partitions, or capacity. Load and fault tests come next, using the traffic and error budgets already stated.

Finish with an operational table: signal, healthy range, action, and user-visible consequence. Include ingestion or queue age, rejected or duplicate work, authoritative-versus-serving divergence, and tail read latency. This makes the proposal inspectable. An interviewer can now change one requirement and see exactly which state, test, and alert must change with it.

Separate what the review establishes from what it leaves open. A passing deterministic fixture proves the declared state transitions for those inputs. It does not establish performance at scale, fairness across real users, or behavior during every network failure. Add a capacity test for the estimated peak, a skew test concentrated on one key or partition, and a fault test at the acknowledgment boundary. Record versions and random seeds so failures can be replayed. When a result differs from expectation, reduce it to the smallest trace, name the violated invariant, and repair that transition before retuning capacity. This evidence-first sequence keeps correctness questions from being hidden inside infrastructure changes.

Keep the review artifact beside the implementation and date the environment that produced it. If a later optimization changes ordering, batching, numeric width, or retry behavior, rerun the same trace before accepting the new result. The invariant is the stable comparison point; the component diagram is not.

A 45-Minute Interview Structure

Spend five minutes on the counter contract and traffic skew. Use the next eight to define APIs, event identity, and the authoritative state. Draw the write path and the striped serving path in twelve minutes. Use ten minutes for the hot-key calculation and exact-versus-stale read choices. Reserve seven minutes for duplicate, cache-loss, regional-partition, and reconciliation scenarios. Close with three minutes of observability: lag, divergence, dedupe, and stripe skew. If requirements change, revise the contract first; do not bolt another component onto a design whose meaning is unclear.

Final Takeaway

A strong distributed counter answer names what is exact, what may lag, and how the displayed number is repaired. Atomic increments solve local races. Event identity solves ambiguous retries. Physical stripes spread a hot logical key. A durable, versioned source of truth makes reconciliation safe. State the remaining tradeoff plainly: cheap, fresh global reads and unlimited write scale cannot all be assumed without coordination or approximation.

Sources and Further Reading

• Redis INCR command
• Redis Cluster scaling and hash slots
• Redis hot-key tracking
• Redis CRDT white paper

Documentation supports the stated system or algorithm behavior; individual question records support only their own practice contracts. Architecture recommendations and interview sequencing are editorial inference.