Articles
Floating-Point Interview Questions: ULPs, NaNs, and Stable Comparisons
Practice floating-point interview questions with ULP neighbor traces, NaN policies, near-zero tolerances, and cancellation cases that expose unstable arithmetic
“Compare floats with epsilon” is an incomplete interview answer. Which epsilon, in which units, at what magnitude, and under what NaN policy? A comparison can pass while the algorithm has already discarded the information that mattered.
This guide separates three decisions: how numbers are represented, what agreement your application requires, and whether the arithmetic preserves a useful result. You will trace adjacent binary64 values, choose tolerances for concrete cases, and diagnose cancellation without hiding it behind a larger threshold.
Evidence boundary: Official language and library behavior is linked where introduced. Goldberg's numerical-analysis paper supplies foundational analysis, rather than candidate testimony. The ledgers and exercises below are original practice material. No candidate reports are used, and no employer-specific interview frequency or format is claimed.
Start with the special-value sliding-window exercise to practice an explicit contract for NaNs and infinities before optimizing the numerical work.
Why can 0.1 look exact but fail an equality test?
Official explanation: Python's floating-point tutorial explains that many decimal fractions have no finite binary representation. On typical binary64 platforms, the stored value is a nearby representable number, while printed output is a decimal representation chosen for readability and round-trip behavior.
In binary64 arithmetic, 0.1 + 0.2 == 0.3 is false. This is not evidence that equality itself is broken. The operations compare their actual represented results, and those results differ. Formatting both sides to two decimal places changes their display; it does not make their original representations identical.
Engineering inference: Ask what the values mean. For exact object identifiers, equality may be appropriate. For measured distances or independently computed numerical results, approximate agreement may be appropriate. For an exact decimal amount, a decimal or scaled-integer representation may better express the contract. None of those choices follows from the word “float” alone.
Distinguish representation error at input from rounding during operations. Converting a value to higher precision after it was rounded cannot reconstruct missing input digits. Similarly, a more accurate summation method cannot recover a small increment already discarded before it reaches the summation routine.
What is an ULP, and why does spacing change?
An ULP is a unit in the last place at a specified scale. For normal binary64 numbers in the interval [2^e, 2^(e+1)), adjacent spacing is 2^(e-52). That formula follows from binary64's 53-bit significand precision, including the leading bit. It is not one fixed absolute epsilon for the whole number line.
Foundational source: Goldberg's paper develops ULP and relative-error reasoning. Below is an original binary64 neighbor ledger, with the format assumption made explicit.
| Starting value | Next larger representable value | Upward gap |
|---|---|---|
1.0 | 1 + 2^-52 | 2^-52 |
2.0 | 2 + 2^-51 | 2^-51 |
2^53 | 2^53 + 2 | 2 |
At 2^53, all nonnegative integers up to that boundary were representable, but the next integer above it is not. Under round-to-nearest, ties-to-even, adding 1.0 to the represented value 2^53 returns 2^53, while adding 2.0 advances to the next value.
A revealing follow-up asks about the neighbor below 1.0. Its gap is 2^-53, half the upward gap. Powers of two sit at spacing transitions. Therefore, dividing every difference by the upward ULP of one endpoint is not a universal way to count representable steps.
Official tools: Python documents math.nextafter and math.ulp. Use the first to inspect neighbors in a specified direction and the second to inspect local spacing. The subnormal region has a different spacing pattern from the normal-number formula; do not extrapolate that formula through zero.
import math
x = 1.0
up = math.nextafter(x, math.inf)
down = math.nextafter(x, -math.inf)
assert up - x == 2.0 ** -52
assert x - down == 2.0 ** -53
Machine epsilon usually describes the gap immediately above one for the format. It does not specify acceptable error for a million-unit measurement or a near-zero residual.
How do NaNs, infinities, and signed zero affect comparisons?
Official language behavior: ECMAScript's strict equality algorithm defines the numeric equality path, including NaN and signed-zero behavior. For ordinary numeric equality, NaN is unequal even to itself, while positive and negative zero compare equal. Equal infinities of the same sign compare equal.
In usual IEEE-style ordered comparisons, a NaN operand is unordered: tests such as less-than, greater-than, and equality are false. A common bug is treating !(x < threshold) as proof that x is at least the threshold. That inference needs a non-NaN precondition.
Engineering inference: Write a policy before implementing an average, sort, or validation routine. Does NaN mean missing input, an invalid computation, or a value that must propagate? Does an infinity invalidate the batch or represent a legitimate unbounded result? Decide whether both signed zeros are equivalent for the application. Some operations preserve their sign even though ordinary equality does not distinguish them.
Do not invent a cross-language rule that every exceptional arithmetic operation produces a float special value. Python's ordinary floating-point division by zero raises an exception; JavaScript Number arithmetic can produce infinity or NaN. The representation model and the language's error behavior are separate contracts.
For approximate comparison, Python's documented math.isclose treats NaN as close to nothing and infinities as close only to themselves. A custom finite-value formula needs separate special-value handling rather than relying on subtraction of infinities.
Choose a tolerance from an actual acceptance contract
Official comparison design: PEP 485 describes Python's symmetric approximate comparison. For finite values, the test uses:
abs(a - b) <= max(rel_tol * max(abs(a), abs(b)), abs_tol)
Relative tolerance scales with magnitude. Absolute tolerance provides a floor in the quantity's own units, especially around zero. The following values are original examples; their thresholds are chosen for the exercise, not recommended defaults for every application.
| Case | Difference and allowed threshold | Decision |
|---|---|---|
1,000,000 versus 1,000,000.5, relative 1e-6 | Difference 0.5; threshold about 1 | Close |
0 versus 1e-12, relative 1e-6, absolute 0 | Difference 1e-12; threshold 1e-18 | Not close |
Same near-zero pair, absolute 1e-11 | Threshold 1e-11 | Close |
The last decision is defensible only if the domain permits that absolute residual. If the values are meters, the absolute threshold is in meters. If they are seconds, it is in seconds. A unit conversion needs a corresponding threshold conversion.
Official library difference: NumPy's isclose uses abs(a-b) <= atol + rtol*abs(b) for finite values. It treats b as the reference and can be asymmetric. It also offers an equal_nan option. Passing the same numeric tolerances to NumPy and Python does not promise identical outcomes.
In an interview, state whether comparison is against a trusted reference or between peers. Explain how tolerances arise from measurement resolution, an algorithmic error budget, or required output accuracy. Choosing a threshold simply because a failing test then turns green proves very little.
Why approximate equality cannot replace ordinary equality everywhere
Original exercise: With absolute tolerance 0.1 and relative tolerance zero, 0 is close to 0.09, and 0.09 is close to 0.18, but 0 is not close to 0.18. Approximate closeness is generally not transitive.
That means it cannot casually replace an equivalence relation used for hashing, grouping, or deduplication. A set of pairwise “close” values may produce order-dependent groups. A comparator that returns equality for nearby values can also violate assumptions needed by ordering algorithms.
Engineering inference: If grouping needs stable identities, define bins or canonical representatives with explicit boundary behavior. Quantization is a different contract from pairwise closeness. Test a value just below and just above a bin edge rather than claiming all nearby values must share a bin.
ULP-distance comparisons can be useful for tightly specified numerical implementations, but they still need rules for NaNs, signs, zero, and the target format. Comparing bit patterns as unsigned integers without handling negative values does not yield a universally meaningful distance.
How do you repair unstable arithmetic rather than hide it?
Foundational analysis: Goldberg distinguishes cancellation that exposes earlier rounding error from subtraction of accurately known values. Subtracting nearby numbers does not automatically make every calculation wrong. The question is which significant information remains after prior operations and input approximations.
Original cancellation case: In binary64, evaluate sqrt(1 + x) - 1 at x = 1e-16. The addition rounds 1 + x to one in the tested round-to-nearest environment, and the direct expression returns zero. Algebraically, for the positive-domain case:
sqrt(1 + x) - 1 = x / (sqrt(1 + x) + 1)
The rewritten expression gives approximately 5e-17 in this exercise. It preserves the small input in the numerator. This is a local example, not a guarantee that the rewrite avoids every overflow, domain, or accuracy issue across all x.
A summation exercise reveals another failure. Perform explicit left-to-right additions of [1e16, 1.0, -1e16]. The middle contribution disappears, so the total is zero. math.fsum returns one in the accompanying check. The Python tutorial explains its more accurate summation approach. Do not use the built-in sum as a permanent synonym for naive repeated addition: its implementation has evolved.
Changing the final comparison tolerance to accept zero as close to one would conceal a lost contribution. Better approaches depend on the operation: accurate summation, algebraic reformulation, scaling, higher precision, or a library routine designed for the numerical problem.
For a test oracle, distinguish the exact intended decimal inputs from the exact values of already-rounded binary inputs. An arbitrary-precision calculation using the wrong input interpretation can confidently validate the wrong question.