Audit Log System Design Interview: Integrity, Search, and Retention

An audit log system design interview asks whether an investigator can trust the history after the system or an administrator is compromised. “Send JSON to Elasticsearch” answers search but not attribution, completeness, tamper evidence, access control, retention, or export verification.

AWS CloudTrail officially documents log-file validation with SHA-256 hashes, signed digest files, and digest chaining. NIST SP 800-92 describes enterprise security log-management processes. These sources establish useful mechanisms and governance concerns. The event envelope, partitioning, and operating model proposed here are editorial inference for the interview requirements.

For a closely related drill, use Design a Compliance Audit Log Platform.

Audit Log Interview: Quick Answer

Applications emit a versioned event with actor, effective subject, action, target, decision, request correlation, event time, receive time, and before/after references. A durable append path assigns sequence or partition position before acknowledgment. Immutable objects are hashed into signed, chained manifests stored under separate administrative control.

Build a redacted search index from the immutable archive. Search authorization is itself audited; tenant and field-level filters run before results are returned. Retention policy attaches to record classes, while legal hold blocks eligible deletion. Verification jobs check signatures, chain continuity, object hashes, ingestion gaps, and index parity.

The Core Contract

An audit log is evidence about security-relevant actions. Its write contract, integrity proof, query authorization, and retention lifecycle must be designed together; a searchable application log is not enough.

Layer	Purpose	Failure it reveals	Trust boundary
event envelope	attribution and semantics	ambiguous actor or target	producer identity
append archive	durable evidence	missing accepted record	ingestion service
signed digest chain	tamper evidence	changed, deleted, reordered batch	signing key/control plane
search index	investigator speed	stale or incomplete view	read authorization
export manifest	portable verification	altered case export	independent verifier

Integrity proof detects unauthorized change; it does not prove the producer reported the truth. Protect producer credentials, record authorization decisions, and compare high-value actions with authoritative domain records.

Design an Event That Can Answer an Investigation

Record stable IDs rather than only display names. Distinguish authenticated actor, delegated subject, service principal, and impersonated user. Include the requested action, authorization result, resource ID and tenant, policy version, request and trace IDs, client context appropriate to the privacy policy, and outcome. Use event time and trusted receive time because client clocks drift.

Avoid dumping secrets or entire payloads. Store classified fields separately or as hashes and references. Schema evolution must preserve old decoding. A registry can reject malformed high-risk events or route them to quarantine, but rejection itself needs an alert because silent loss destroys completeness.

Make Accepted Batches Tamper-Evident

Write events to an append-only durable stream, compact them into objects, and calculate a cryptographic hash for each object. A manifest lists object hashes and the previous manifest hash, then a protected signing service signs it. Replicate archive objects and manifests to an account or principal that ordinary application administrators cannot rewrite.

Hash chaining narrows the evidence problem: an attacker who removes or changes a batch breaks continuity or a signature. It does not stop an authorized producer from emitting a false event. Key rotation needs signed transition metadata, and verifier tooling must retain the public keys and algorithms needed for old periods.

Separate Search Convenience from Evidence

Index selected fields for time, actor, resource, action, outcome, and correlation searches. Treat the index as a rebuildable projection. Its document includes archive object and offset so an investigator can retrieve the source event and verify it. Search lag, dropped-document counts, and archive-index cardinality differences are explicit health signals.

Enforce tenant filters before query execution, not after receiving a broad result. Restrict sensitive fields by role and purpose; cap exports and watermark them with case, requester, and time. Log query, view, and export actions into the audit system through an isolated path to avoid a trivial recursive storm.

Apply Retention Without Destroying Holds

Map each event class to a policy version with minimum/maximum retention, residency, and permitted deletion method. A legal or investigation hold references immutable object ranges or event IDs and prevents the lifecycle worker from deleting them. Removing a hold requires an independently audited action.

Deletion is asynchronous: mark eligible objects, remove search projections, delete or cryptographically erase archive material, and produce a completion receipt. Backups follow their own lifecycle; do not promise immediate byte removal if the platform only expires backup media later. Periodic restore-and-verify exercises prove that retained evidence remains usable.

Work Through a Failure Trace

Create three archive batches with sequences 1–300 and chained manifests. Delete batch 2, modify one event in batch 3, and remove a search document from batch 1. The verifier should report a missing object/chain break, a hash mismatch, and index parity drift as three separate failures. An unauthorized cross-tenant query must return no rows and emit a denied-access audit event.

Audit Log check: Which actions and actors must be attributable? Walk the trace once in event-time order, once with retries and reordering, and once after rebuilding projections from durable records. State the expected invariant after every step. If the outcome depends on message arrival order when the contract says it should not, the design is missing an identity, version, or transaction boundary. If the design intentionally permits a temporary difference, give it a watermark, maximum age, and repair owner.

Audit Log check: How is a missing, reordered, or altered record detected? This trace is an interview model, not a measured production benchmark. Add a skew test concentrated on one tenant or key, a two-times-peak capacity test, and a fault injection at the acknowledgment boundary. Preserve input versions and random seeds so a failed run can be reduced and replayed.

Turn the Requirements into Verification

Write these five questions beside the diagram: Which actions and actors must be attributable? How is a missing, reordered, or altered record detected? Who may search sensitive audit evidence? How do retention and legal holds affect deletion? Can investigators verify exported results independently? Each question must point to a record, transition, test, and observable signal. If one can be answered only with 'the service handles it,' the contract is still incomplete.

Audit Log check: Who may search sensitive audit evidence? Build a deterministic fixture with five to ten timestamped inputs. Include an identical retry, a conflicting reuse of an ID, a late or stale input, a worker failure after durable work but before acknowledgment, and a rebuild of the serving projection. Expected results must be written independently of the implementation. Verify the authoritative state, the serving state through a named watermark, and every emitted correction or rejection reason.

Audit Log check: How do retention and legal holds affect deletion? Then add property checks. Applied logical IDs remain unique; versions never move backward; reserved or consumed value obeys the topic's invariant; a rebuild through the same input boundary produces the same digest; and an unauthorized reader cannot cross its scope. Generate reordered deliveries around the boundary instead of testing only the happy sequence. A passing fixture proves these declared cases, not production throughput or every network failure.

Audit Log check: Can investigators verify exported results independently? For fault testing, stop a worker immediately before and after its commit, duplicate queue delivery, delay one partition, and make the read projection unavailable. The runbook should say whether intake pauses, work accumulates durably, readers serve a last-known-good generation, or the request fails closed. Record the recovery point and compare the rebuilt digest before switching traffic. This is concrete evidence that the recovery story matches the write contract.

Capacity, Operations, and Recovery

Audit Log check: Which actions and actors must be attributable? Estimate the durable write rate, average and peak payload, partition skew, retention volume, read or query shape, and acceptable lag. Convert those assumptions into partitions, worker concurrency, storage tiers, and backpressure. Numbers are sizing inputs, not universal facts. Say which limit fails first and what the user sees when it does.

Audit Log check: How is a missing, reordered, or altered record detected? Operate from contract metrics: accepted-versus-applied counts, duplicate/conflict rate, oldest pending work, authoritative-versus-serving divergence, retry exhaustion, and p95/p99 latency. Alert on sustained error-budget consumption and correctness gaps rather than raw traffic. Keep a runbook that can pause intake or rollout, drain work, rebuild a named projection through watermark W, verify a checksum, and switch readers to the new generation.

Audit Log check: Who may search sensitive audit evidence? Use least privilege between producers, workers, control plane, data plane, and investigators. Encrypt transit and storage, rotate credentials, audit high-risk actions, and avoid placing secrets or unnecessary personal data in operational logs. Security controls support the data contract; they do not replace idempotency or verification.

Misleading Answers and Better Follow-Ups

Audit Log check: How do retention and legal holds affect deletion? A weak answer names Kafka, Redis, and a database before defining truth. Another says “exactly once” without identifying the idempotency key, persistence boundary, or replay behavior. A third adds a cache but cannot explain invalidation or rebuild. Replace each slogan with one invariant, one linearization point, and one counterexample.

Audit Log check: Can investigators verify exported results independently? Ask the interviewer to change a requirement: permit temporary staleness, add multiple regions, require a correction after finalization, or remove a dependency. Explain which state transition, key, and monitor changes. That is more useful than redrawing the same boxes at a larger scale.

Audit Log check: Which actions and actors must be attributable? Separate three evidence classes in the answer. Official documentation supports the cited product or protocol behavior. The proposed architecture, thresholds, and sequencing are inference from the stated requirements. This prevents a sensible design choice from being presented as an external fact.

A 45-Minute Interview Plan

Spend the first five minutes clarifying actors, scale, correctness, retention, and failure expectations. Use the next eight to define the authoritative records, IDs, versions, and invariants. Draw the write path and its acknowledgment boundary for ten minutes, then the read or serving path for five.

Use ten minutes for the title-specific failure trace, including one retry, one stale or late event, and one recovery. Reserve the final seven minutes for sizing, skew, access control, monitors, and trade-offs. Keep a short list of assumptions visible. When the interviewer changes one, update the affected contract before changing components.

End with the source of truth, the permitted temporary inconsistency, and the repair procedure. A compact design that can be replayed and verified is stronger than a large architecture with undefined states.

Final Takeaway

For a audit log system design interview, lead with this thesis: An audit log is evidence about security-relevant actions. Its write contract, integrity proof, query authorization, and retention lifecycle must be designed together; a searchable application log is not enough. Define the evidence boundary, run a concrete race or failure trace, and show how the system detects and repairs divergence.

Sources and Further Reading

• AWS CloudTrail log-file validation
• NIST SP 800-92
• NIST Digital Identity Guidelines

Official sources support only the cited mechanism. Architecture recommendations, capacity figures, and interview sequencing are editorial inference.