Property-Based Testing Interview Questions: Invariants, Generators, and Shrinking

Property-based testing interview questions test whether you can describe correctness beyond a handful of examples. Generating thousands of inputs is useful only when the assertion rejects meaningful mistakes. A deduplication function that always returns an empty list satisfies “the output contains no duplicates,” yet loses every value the caller supplied.

Explain the contract, the input domain, and the failure you expect the harness to expose before choosing a generator or a test budget.

Evidence boundary: Hypothesis documentation establishes the library behavior discussed here. The stable-deduplication exercise, property ledger, and shrinking trace are original examples, not a recorded Hypothesis run or an employer's official interview rubric. Related candidate-reported question records provide practice contexts without establishing which testing library an interviewer used.

A property must rule out a plausible wrong implementation

Official concept: Hypothesis combines test functions with strategies that generate inputs. It can explore many cases and report failures, while example-based tests remain useful alongside generated tests. Hypothesis: introduction.

For this original exercise, define stable_unique(values) as follows: given a list of integers, return each distinct value once, in the order of its first appearance, without changing the input list. For [4, 1, 4, 2, 1], the result must be [4, 1, 2].

The word “stable” carries a requirement. sorted(set(values)) removes duplicates but changes first-occurrence order. An empty list also contains no duplicates. Returning the original input preserves membership and order but fails to remove repeated values.

Turn those mistakes into a property ledger:

Contract property	Wrong implementation it can expose	Why it is insufficient alone
Output has no duplicates	Return the input unchanged	Returning an empty list still passes
Input and output contain the same distinct values	Drop a value or add an invented value	A sorted set still passes
Output follows first-occurrence order	Sort the distinct values	A correctly ordered subset may still lose values
Input list remains unchanged	Sort or remove items in place	An incorrect new output may leave input untouched
Applying the function twice changes nothing	Reintroduce unstable transformations	Always returning an empty list also passes

The first three properties jointly characterize the returned values for this domain. Input preservation specifies the side-effect contract. Idempotence is a useful additional relationship, but it does not replace membership and order checks.

Write the oracle separately from the generator

Here is an original Python contract checker. The integer-only domain makes values hashable and equality straightforward. If the real API accepts records, floating-point special values, or custom equality, define those semantics separately.

def assert_stable_unique_contract(values, result):
    assert len(result) == len(set(result))
    assert set(result) == set(values)
    first_positions = [values.index(value) for value in result]
    assert first_positions == sorted(first_positions)

This checker uses a deliberately simple, potentially quadratic approach to finding first positions. For short test inputs, clarity matters more than matching the production algorithm's efficiency. An independent oracle should be easier to inspect than the implementation it evaluates.

A compact reference implementation for the stated integer domain is list(dict.fromkeys(values)). Official Python behavior: dictionaries preserve insertion order. That supports this reference, but the contract checker above expresses the requirement independently of using a dictionary. Python: dictionary behavior.

In an interview, explain whether an oracle is independent. Copying the production algorithm into the test can reproduce its bug. Comparing against a trusted library or a slower specification can be useful, but you still need confidence that its semantics match the actual contract.

Generate the full relevant domain, including inconvenient cases

Official guidance: Hypothesis separates the domain a strategy can generate from the distribution of examples it chooses. Its guidance emphasizes selecting a sufficiently general domain rather than treating input frequencies as a production traffic model. Hypothesis: domain and distribution.

For stable deduplication, empty lists, singletons, repeated values, negative values, and differently ordered distinct values are all legitimate inputs. A generator restricted to already unique ascending lists hides two important bug classes: failure to remove duplicates and accidental sorting.

A Hypothesis test can connect the checker to generated lists as follows. The module import represents the implementation under test; place the contract helper above in the same test file.

from hypothesis import given, strategies as st
from my_module import stable_unique

@given(st.lists(st.integers(), max_size=60))
def test_stable_unique(values):
    original = values.copy()
    result = stable_unique(values)
    assert values == original
    assert_stable_unique_contract(original, result)
    assert stable_unique(result) == result

Official API: lists composes an element strategy with size constraints. The 60-item limit here is a test-runtime choice, not a claim that the API accepts only 60 items. Hypothesis: strategies reference.

Keep explicit larger-input or performance tests if size matters to the contract. Property-based functional testing and scalability testing answer different questions. Passing many short lists does not establish that a service can process a million records within its deadline.

Construct valid inputs instead of rejecting almost everything

Suppose an interview exercise requires two intervals with the second beginning after the first ends. Independently generating four arbitrary endpoints and rejecting invalid combinations may waste much of the search budget. Generate a start, a nonnegative length, and a nonnegative gap, then derive consistent endpoints.

Official behavior: Hypothesis supports adapting strategies with transformations and filtering. assume() can reject an entire test case; excessive filtering can trigger a health check and reduce useful exploration. Hypothesis: adapting strategies.

For the deduplication exercise, do not filter out lists containing duplicates just because a failing example is awkward. Those inputs are central to the requirement. If the implementation rejects some values by contract, test that rejection separately from the valid-input properties.

A useful generator review asks two questions: Can it produce the important failure-triggering shapes? Can it produce enough valid cases efficiently? A beautifully concise strategy that excludes every difficult boundary is weaker than a slightly longer strategy with an honest domain.

Shrinking turns a failure into a smaller explanation

Official behavior: Hypothesis strategies support simplification of generated failures. The structure of the strategy affects what can be simplified; generation constraints need to remain meaningful during this process. Hypothesis project: strategies that shrink.

For the incorrect implementation sorted(set(values)), imagine a failing input [9, 2, 9, 4]. Removing irrelevant elements can leave [9, 2]. Simplifying the values while preserving descending first-occurrence order can produce [1, 0]. The correct result is [1, 0]; the broken implementation returns [0, 1].

This is an illustrative simplification trace, not a promise that Hypothesis will take these exact steps or return this exact pair. The useful result is a small valid counterexample that still demonstrates the same problem. Do not claim a globally smallest counterexample without a guarantee appropriate to the tool and domain.

Notice what the small example teaches: duplicates were not necessary to expose this particular sorting error. The essential condition is that first-occurrence order differs from numeric order. Shrinking can clarify the cause rather than merely make the input shorter.

If shrinking gets stuck in large cases, inspect the generator and the test's dependencies. Does removing one record invalidate every reference? Can a simpler valid graph be constructed? Is each run observing the same state? Generator structure is part of the debugging design.

Reproduce failures without depending only on a seed

Official behavior: Hypothesis can save and replay failures through its example database and supports explicit @example cases. Its documentation warns that database entries and reproduction blobs are not permanent correctness guarantees across changes; an explicit regression example is the durable choice for an important case. Hypothesis: replaying failed tests.

For the sorting bug, preserve [1, 0] as a small regression example with a name explaining first-occurrence order. Keep the general property as well, so future exploration continues beyond that one case.

Record the implementation revision, library version, test configuration, and failing input when investigating CI. A seed alone does not freeze external services, current time, filesystem contents, or a changed generator. Treat reproduction as an environment and input problem together.

Official caution: unstable external state can prevent Hypothesis from reliably replaying and shrinking a failure. Examples include globals, database state left between cases, and unmanaged randomness that changes the result. Hypothesis: flaky failures.

Reset fixtures between generated cases and use a controllable clock where timing is part of the model. Do not silence a flaky-failure signal merely to make the run green; first determine which observation changes when the same input is replayed.

Stateful properties test sequences, not just isolated values

Official capability: Hypothesis stateful testing generates actions as well as values. Rules, preconditions, and invariants let a test compare an evolving system with a model across a sequence of operations. Hypothesis: stateful tests.

Extend the topic to a notification deduplication service. The original contract might say that repeating the same key is suppressed for ten minutes after its first accepted delivery. Before generating anything, clarify whether a suppressed duplicate extends that window and what happens exactly at expiry.

Use a fake clock and a simple model mapping keys to expiry times. Generate operations such as deliver a key, repeat a key, and advance time. Compare each acceptance decision with the model, including a repeat just before expiry and at the boundary.

A failing sequence might shrink to three actions: accept key A, advance to expiry, deliver A again. That sequence distinguishes an incorrect inclusive comparison from the agreed boundary rule. If the test always creates a fresh key, it never exercises deduplication even though it sends many requests.

Make cleanup part of each generated sequence. Leaked keys from earlier examples can turn a valid first delivery into an unexpected duplicate and make the failure dependent on execution order rather than the reported sequence.

Round-trip tests also need an independent anchor. A serializer and parser can agree with each other while both violating an external format. If one writes an incorrect field name and the other expects that same name, an internal round trip may pass. Add a few contract examples from the documented format or compare against an independent implementation when appropriate.

For the notification service, successful model agreement is bounded by the operations you generate. A sequential state machine does not automatically explore concurrent delivery races. State that limitation and add a separate concurrency test when two workers can accept the same key simultaneously. A property is strongest when its scope is explicit: it can establish a useful invariant within the explored model without pretending to cover every execution the real system permits.

Sources and Further Reading

• Hypothesis: introduction
• Python: dictionary insertion order
• Hypothesis: domain and distribution
• Hypothesis: strategies reference
• Hypothesis: adapting strategies
• Hypothesis project: strategies that shrink
• Hypothesis: replaying failures
• Hypothesis: flaky failures
• Hypothesis: stateful tests