Online Judge System Design Interview: Test Cases, Scoring, and Reproducibility

An online judge system design interview is not only about running untrusted code. The sharper question is whether the same submission, problem version, environment, and scoring rules produce an explainable verdict today and during a rejudge next week. If test files or compiler images mutate in place, neither contestants nor operators can reproduce the result.

DOMjudge officially documents pass/fail and partial scoring, testcase groups, custom validators, and rejudging workflows. Linux documents cgroup resource-control primitives. This article keeps isolation at the boundary and focuses on the title's distinct problem: immutable tests, scoring, replay, and reproducibility. Architecture choices beyond the documentation are editorial inference.

For a closely related drill, use Debug a Concurrent Job Scheduler.

Online Judge Interview: Quick Answer

On submission, create an immutable judging manifest that pins source hash, language/compiler image digest, problem and testcase bundle version, validator digest, time/memory limits, scoring policy, and random seed. Queue a job by manifest ID. A worker runs each case in a fresh sandbox, uploads stdout/stderr excerpts and resource measurements, and writes idempotent case results.

An aggregator calculates the verdict and score from the pinned policy. Publishing is a compare-and-set from judging generation to completed. Rejudging creates a new generation with a new manifest; it never overwrites the old evidence. The scoreboard points to the active generation and can be rebuilt.

The Core Contract

An online judge is a versioned evaluation ledger. Reproducibility depends on binding source, compiler, sandbox, tests, validator, limits, and scoring rule into one immutable judging manifest.

Pinned input	Why it matters	Example drift if omitted
source hash	identifies submitted program	mutable upload
compiler image digest	fixes toolchain and libraries	new optimizer changes behavior
testcase bundle version	fixes inputs and answers	repaired hidden case
validator digest	fixes comparison semantics	whitespace or tolerance change
scoring policy	fixes group weights and dependencies	same cases, different score
limits and seed	fixes execution contract	flaky randomized judge

The manifest is the unit of replay. A database row that says only “Accepted” is not enough evidence.

Version the Whole Problem Contract

A problem release contains statement, attachments, test inputs, expected outputs or validator, group membership, weights, dependencies, and limits. Store content-addressed blobs and an immutable release manifest. Draft edits create a new version. Contest configuration chooses exactly one release before opening.

Hidden tests remain inaccessible to contestants, but authorization is separate from versioning. Workers receive short-lived access to the selected bundle. Logs redact hidden input and expected output while preserving hashes and bounded diagnostics for operators. Encrypting files does not fix a service that grants broad worker or administrator access, so name the roles.

Make Case Execution Idempotent and Diagnosable

Use the key (submission_id, judging_generation, testcase_id, attempt). A worker lease has a deadline and heartbeat. If it disappears, another worker may retry, but only one accepted result wins for the generation. Upload result artifacts before committing their references. Distinguish compilation error, wrong answer, time limit, memory limit, output limit, runtime error, validator failure, and infrastructure failure.

Infrastructure failure should generally retry without penalizing the submission. User-code nondeterminism may produce different results; retain attempts and flag divergence. Pin CPU architecture where it matters, normalize locale and timezone, control environment variables, and capture kernel/runtime versions.

Calculate Scores from an Explicit Policy

For pass/fail, all required cases must pass. Partial scoring may sum case weights, award a group only if all its cases pass, or call a custom validator that returns a bounded score. Encode the rule as data and validate weights. If group B depends on A, decide whether B runs for diagnostics or is skipped after A fails.

The aggregator is a pure function of accepted case results and policy version. Store its input hash and output. Scoreboard caches consume completed generations, and replaying the same generation must not double-apply penalties or submissions. This is where idempotency matters beyond workers.

Rejudge Without Erasing History

A faulty expected answer may affect thousands of submissions. Freeze the bad problem release, create a corrected release, select the affected submission set, and enqueue a new generation. Keep old and new outcomes, reason, initiator, and activation time. Compute impact before switching the scoreboard pointer.

Roll out rejudging in cohorts and compare verdict transitions. Stop if infrastructure errors or unexpected churn exceed the stated threshold. Notify users according to contest policy. A rollback changes the active generation pointer; it does not delete the corrected or original evidence.

Work Through a Failure Trace

Submission S is judged with problem v7, compiler digest C, validator V3, and tests T7. Cases 1–2 pass, group 2 fails one case, and one worker lease expires. The retry uses the same manifest and case ID; the late first result cannot overwrite the accepted attempt. Problem v8 fixes the case and creates generation 2. Both 70 and 100 scores remain explainable.

Online Judge check: How are hidden tests and validators versioned? Walk the trace once in event-time order, once with retries and reordering, and once after rebuilding projections from durable records. State the expected invariant after every step. If the outcome depends on message arrival order when the contract says it should not, the design is missing an identity, version, or transaction boundary. If the design intentionally permits a temporary difference, give it a watermark, maximum age, and repair owner.

Online Judge check: What makes a verdict reproducible across workers? This trace is an interview model, not a measured production benchmark. Add a skew test concentrated on one tenant or key, a two-times-peak capacity test, and a fault injection at the acknowledgment boundary. Preserve input versions and random seeds so a failed run can be reduced and replayed.

Turn the Requirements into Verification

Write these five questions beside the diagram: How are hidden tests and validators versioned? What makes a verdict reproducible across workers? How are partial scores and testcase groups calculated? What happens when a bad testcase requires rejudging? How are worker timeouts distinguished from wrong answers? Each question must point to a record, transition, test, and observable signal. If one can be answered only with 'the service handles it,' the contract is still incomplete.

Online Judge check: How are partial scores and testcase groups calculated? Build a deterministic fixture with five to ten timestamped inputs. Include an identical retry, a conflicting reuse of an ID, a late or stale input, a worker failure after durable work but before acknowledgment, and a rebuild of the serving projection. Expected results must be written independently of the implementation. Verify the authoritative state, the serving state through a named watermark, and every emitted correction or rejection reason.

Online Judge check: What happens when a bad testcase requires rejudging? Then add property checks. Applied logical IDs remain unique; versions never move backward; reserved or consumed value obeys the topic's invariant; a rebuild through the same input boundary produces the same digest; and an unauthorized reader cannot cross its scope. Generate reordered deliveries around the boundary instead of testing only the happy sequence. A passing fixture proves these declared cases, not production throughput or every network failure.

Online Judge check: How are worker timeouts distinguished from wrong answers? For fault testing, stop a worker immediately before and after its commit, duplicate queue delivery, delay one partition, and make the read projection unavailable. The runbook should say whether intake pauses, work accumulates durably, readers serve a last-known-good generation, or the request fails closed. Record the recovery point and compare the rebuilt digest before switching traffic. This is concrete evidence that the recovery story matches the write contract.

Capacity, Operations, and Recovery

Online Judge check: How are hidden tests and validators versioned? Estimate the durable write rate, average and peak payload, partition skew, retention volume, read or query shape, and acceptable lag. Convert those assumptions into partitions, worker concurrency, storage tiers, and backpressure. Numbers are sizing inputs, not universal facts. Say which limit fails first and what the user sees when it does.

Online Judge check: What makes a verdict reproducible across workers? Operate from contract metrics: accepted-versus-applied counts, duplicate/conflict rate, oldest pending work, authoritative-versus-serving divergence, retry exhaustion, and p95/p99 latency. Alert on sustained error-budget consumption and correctness gaps rather than raw traffic. Keep a runbook that can pause intake or rollout, drain work, rebuild a named projection through watermark W, verify a checksum, and switch readers to the new generation.

Online Judge check: How are partial scores and testcase groups calculated? Use least privilege between producers, workers, control plane, data plane, and investigators. Encrypt transit and storage, rotate credentials, audit high-risk actions, and avoid placing secrets or unnecessary personal data in operational logs. Security controls support the data contract; they do not replace idempotency or verification.

Misleading Answers and Better Follow-Ups

Online Judge check: What happens when a bad testcase requires rejudging? A weak answer names Kafka, Redis, and a database before defining truth. Another says “exactly once” without identifying the idempotency key, persistence boundary, or replay behavior. A third adds a cache but cannot explain invalidation or rebuild. Replace each slogan with one invariant, one linearization point, and one counterexample.

Online Judge check: How are worker timeouts distinguished from wrong answers? Ask the interviewer to change a requirement: permit temporary staleness, add multiple regions, require a correction after finalization, or remove a dependency. Explain which state transition, key, and monitor changes. That is more useful than redrawing the same boxes at a larger scale.

Online Judge check: How are hidden tests and validators versioned? Separate three evidence classes in the answer. Official documentation supports the cited product or protocol behavior. The proposed architecture, thresholds, and sequencing are inference from the stated requirements. This prevents a sensible design choice from being presented as an external fact.

A 45-Minute Interview Plan

Spend the first five minutes clarifying actors, scale, correctness, retention, and failure expectations. Use the next eight to define the authoritative records, IDs, versions, and invariants. Draw the write path and its acknowledgment boundary for ten minutes, then the read or serving path for five.

Use ten minutes for the title-specific failure trace, including one retry, one stale or late event, and one recovery. Reserve the final seven minutes for sizing, skew, access control, monitors, and trade-offs. Keep a short list of assumptions visible. When the interviewer changes one, update the affected contract before changing components.

End with the source of truth, the permitted temporary inconsistency, and the repair procedure. A compact design that can be replayed and verified is stronger than a large architecture with undefined states.

Final Takeaway

For a online judge system design interview, lead with this thesis: An online judge is a versioned evaluation ledger. Reproducibility depends on binding source, compiler, sandbox, tests, validator, limits, and scoring rule into one immutable judging manifest. Define the evidence boundary, run a concrete race or failure trace, and show how the system detects and repairs divergence.

Sources and Further Reading

• DOMjudge scoring documentation
• DOMjudge administrator manual
• Linux cgroup v2 documentation

Official sources support only the cited mechanism. Architecture recommendations, capacity figures, and interview sequencing are editorial inference.