Build Cache System Design Interview: Content Keys, Hermeticity, and Unsafe Cache Hits

A remote build cache can return the wrong binary without a hash collision, network error, or corrupted file. If the action key omits the compiler version, two different builds can legitimately look identical to the cache. A faster incorrect build is still incorrect.

Build cache system design interviews reward a clear reuse contract: identify all inputs that affect the action, execute under that contract, and accept results only from an appropriate trust boundary. Start with Design CI/CD Build Caching and challenge every proposed cache hit.

Evidence boundary: Bazel's official documentation describes remote caching and hermeticity. The omitted-toolchain incident, key format, trust design, and verification exercises here are original scenarios and architectural recommendations. They are not reports of a company's interview questions or production incidents.

What does a build cache actually map?

Distinguish an action cache from content-addressable storage. In a proposed design, the action cache maps an action digest to a result manifest. The manifest references output blobs by their content digests and records exit status and other required result information. The blob store maps content digests to bytes.

Bazel's remote caching documentation describes action metadata and content-addressed outputs as separate parts of the cache. That distinction matters because an action result can reference several stored outputs. The schema and trust rules below are design choices for this interview exercise. Bazel remote caching.

A blob digest answers “Are these the bytes named by this digest?” It does not answer “Were these bytes produced by the intended action?” A malicious or mistaken writer can map an action key to a valid blob containing the wrong result. Both layers need validation according to the threat model.

A source-commit hash is usually too coarse in one direction and too weak in another. It can invalidate unrelated actions when any file changes, yet fail to identify an external toolchain or environment difference. Think in terms of declared action inputs rather than only repository identity.

Build a complete and unambiguous action key

An illustrative action identity includes command and arguments, declared environment, working-directory semantics, input tree digest, toolchain identity, execution platform, and output declarations. Include the version of any rule or key schema whose change can affect execution semantics.

Serialize fields canonically. Concatenating ab and c produces the same text as a and bc; a typed, length-delimited encoding avoids that ambiguity. Sort sets whose order is semantically irrelevant, but preserve argument order and other order-sensitive fields.

For a directory input, hash its logical tree: paths, file content identities, and metadata that the execution contract considers meaningful. Executable bits and symlink handling may affect behavior. Absolute host paths should not accidentally become semantic inputs unless the build depends on them.

Do not put secrets themselves into broadly visible keys or manifests. Where secret-dependent execution is unavoidable, define a protected identity/version scheme and access policy, or mark the action non-cacheable. Hashing a low-entropy secret is not a safe way to publish it.

An action's dependencies include generated files. The consumer key must include the generated artifact identity, and the producer must itself have a correct key. A downstream content hash can preserve the wrong generated bytes perfectly if the upstream action hit was unsafe.

Trace an unsafe hit caused by a missing compiler

This original scenario compiles source S with compiler A, producing artifact OA. The cache key is only H(source S, flags). A developer switches to compiler B, which produces OB under the intended execution contract. The cache returns OA because the key did not change.

There is no need for a cryptographic collision. The key function intentionally mapped two different actions to the same identity by leaving out a relevant input. Verifying OA's blob digest confirms its integrity, not its suitability for the second build.

Changed condition	Required cache decision
Only an unrelated source file changes	Reuse this action if its declared inputs are unchanged.
Compiler changes from A to B	Miss unless the action contract explicitly proves equivalence.
Required environment flag changes	Key must change or the action must not be cacheable.
Cache returns a missing output blob	Treat the result as unusable; rebuild or recover it safely.

Repair both the action identity and affected cache namespace. Simply deleting one entry is temporary if clients keep emitting the incomplete key. Roll out a new key schema or toolchain digest and explain how old writers are prevented from reintroducing unsafe results.

For an interview, name the causal chain aloud: omitted input, equal action key, reused old output, incorrect behavior. That is more useful than vaguely calling the incident “stale cache” and choosing a shorter TTL.

Why does hermeticity matter beyond hashing?

Bazel describes hermetic builds as isolated from changes to host tools and libraries, with explicit tool and dependency identities. This official concept supports the reuse contract; your implementation still has to enforce what the action can observe. Bazel hermeticity.

If a compiler reads an undeclared header from the host filesystem, a complete-looking key remains incomplete. If a test fetches a changing URL, recorded source files do not fully determine its result. A sandbox, controlled toolchain, and network policy reduce these hidden inputs.

Hermeticity and determinism are related but distinct concerns. An action can have controlled inputs yet use an uncontrolled random seed or emit timestamps. Conversely, an apparently repeatable local action may depend on an ambient library that simply has not changed yet.

Some outputs intentionally vary. You can normalize irrelevant metadata only if the build contract permits it; removing a timestamp from a signed artifact may change meaningful bytes. Do not equate “different digest” with a bug until the expected equivalence relation is defined.

A practical migration begins by finding actions with undeclared files, host binaries, network reads, timezone dependence, and unstable ordering. Disable shared caching for actions whose reuse contract is unknown. Correctness should precede expanding the writer fleet.

Separate trusted writers from ordinary readers

Bazel also distinguishes remote execution, which runs actions on remote workers, from caching results. Moving execution does not remove the need to identify its inputs.

A recommended production policy lets controlled CI workers publish shared action results while developers read them. Developer writes can use a separate namespace until their execution context and identity meet the same requirements. This is a trust design option, not a claim that all teams must use one setup.

Namespace isolation matters for both integrity and confidentiality. A tenant who knows another tenant's digest must not automatically retrieve private source or outputs. Read authorization should be enforced even for immutable blobs and should not rely on obscure content addresses.

The writer identity should be attached to the action-result mapping through authenticated publication or verifiable provenance where needed. A signature proves that a particular identity signed a statement; it does not prove the worker executed the action correctly. Define what makes that identity trusted and how compromise is handled.

Separate untrusted pull-request builds from release-cache writers. Even if a pull request cannot change the cache server, it may modify build scripts executed by a privileged worker. Isolation, scoped credentials, and restricted publication close that route to poisoning.

On suspected poisoning, block the affected writer or namespace, preserve evidence, and rebuild relevant artifacts from trusted inputs. Blindly warming the cache again with the same compromised workers repeats the failure.

Publish results without dangling references

Upload and verify output blobs before making the action result discoverable. Then commit the result manifest that references those durable objects. A process crash may leave unreferenced blobs, which are a storage cleanup problem; publishing a manifest before its outputs exist creates a correctness problem for readers.

Duplicate writers can race on the same action key. If they produce identical result manifests, the operation can converge harmlessly. If they produce different outputs for a supposedly deterministic action, record the conflict and quarantine or reject it under the stated policy. Last-writer-wins hides evidence of nondeterminism or compromise.

Large outputs may use chunking, but the result needs a complete verified manifest. Readers should not observe an incomplete artifact as successful merely because some chunks downloaded. Temporary local files should be atomically promoted after verification according to the client filesystem contract.

Errors and failures need explicit caching rules. A deterministic compiler rejection may be cacheable in a specified environment, while a network timeout is not equivalent to an action's semantic failure. Decide which exit conditions can be reused and how long their diagnostic context remains useful.

What should eviction and cache outages do?

For an ordinary recomputable build cache, a miss falls back to execution. Missing blobs invalidate the corresponding cached result for that request; they should not become mysterious build failures when the system can rebuild. Release archives with mandatory retention are a different product and should not rely on disposable cache policy.

Garbage collection can mark referenced objects from retained action manifests, then remove unreferenced blobs after a grace period. Concurrent upload and read operations need protection from collection races. Pin active publication generations or use a protocol that makes liveness unambiguous.

A hot cache is not automatically worth its cost. Compare bytes transferred and lookup latency with execution time saved. A tiny action may run faster locally than downloading its outputs. A large artifact may overwhelm the network even when its computation is expensive.

Avoid a cache outage causing a thundering herd of identical rebuilds. Local coalescing, bounded fallbacks, and retry budgets can reduce pressure. Keep these availability controls distinct from the identity rules: no amount of request deduplication repairs an unsafe key.

How do you demonstrate correctness and value?

Create a small controlled action whose output changes with compiler identity, an environment flag, and a declared input. Verify that each meaningful change misses. Then change an unrelated file and verify reuse where the action contract allows it. These tests exercise the boundary rather than merely checking that GET returns bytes.

Rebuild a sample of cache hits independently in trusted environments and compare outputs under the intended equivalence rule. This is evidence about sampled actions, not proof that every historical cache entry is safe. Record mismatches by rule, platform, writer, and key version.

Measure avoided execution time, transfer cost, hit latency, missing-blob rate, conflicting-result rate, and sampled rebuild mismatch rate. Hit ratio alone can increase when a broken key collapses unrelated actions. In that case, a higher ratio is evidence of danger rather than improvement.

A strong design answer ends with an invariant: “A shared hit is reusable only when action identity is complete, execution respects that identity, and the result mapping comes from an authorized trust context.” Then show the missing-compiler counterexample and the publication crash boundary.

Practise safe reuse across the build pipeline

Two overlap with the registry article because digest-addressed artifact storage and dependency graphs are shared components; the three other prompts shift the exercise to execution identity and CI trust.

Question	Practice focus
Design CI/CD Build Caching	Explain action keys and unsafe reuse.
Design a Distributed Blob Storage System for Build Artifacts	Separate result mappings from stored blobs.
Design a Dependency-Aware CI/CD Pipeline	Include generated dependencies in action identity.
Design Google-scale CI/CD pipeline	Choose trusted writers and execution boundaries.
Docker optimization, container fundamentals, and CI/CD for an ML model service	Identify ambient tools and container differences.

Start with build caching. Deliberately omit one dependency from your proposed key, explain the resulting unsafe hit, and repair the execution contract as well as the lookup function.

Sources and Further Reading

• Bazel: Remote caching

• Bazel: Hermeticity

• Bazel: Remote execution overview

Primary references checked September 30, 2026. Product details and supported versions can change; the worked examples are original practice material.