Backtesting Interview Questions: Diagnose Leakage, Selection Bias, and Unrealistic Returns

Backtesting interview questions test whether you can reconstruct what was knowable, what was tradable, and what was selected after looking at results. A smooth return curve is the output of those assumptions. Before discussing Sharpe ratio, audit the timestamps, historical universe, execution model, costs, and research history that produced it.

This guide uses QuantConnect's official documentation and primary research on backtest overfitting. The timestamps, portfolios, costs, and calculations are original educational examples. They are not measured investment returns, employer interview reports, or recommendations to trade a strategy.

What would you ask before accepting a backtest?

Ask for the hypothesis, instrument universe, decision frequency, input data, execution assumptions, and evaluation period. Then ask how many configurations were tried. These are our audit recommendations: the answers determine whether the reported result describes a feasible strategy or a favorable reconstruction of history.

Suppose a researcher reports an attractive daily strategy using company announcements. A useful first response is not “add a bigger test set.” Establish when announcements became available, when signals were computed, which price was used, and whether the researcher changed the rule after inspecting evaluation results.

Keep an assumption ledger with an owner and an observable check for each entry. “Data is point-in-time” should point to version and availability fields. “Trading is realistic” should point to an order and fill record. “The holdout is untouched” should point to a research log. Labels alone cannot establish these properties.

QuantConnect's research guide identifies look-ahead and survivorship bias as research hazards. That official platform guidance supports checking information and sample construction; the specific ledger and numerical examples here are our own. Official research guide.

Which timestamp makes information usable?

Separate the time an event concerns from the time its information becomes available. A financial record for an earlier quarter can be published later, revised later still, and arrive in your data feed after publication. Using its final value at the quarter's end can place future information into an earlier decision.

Our synthetic announcement concerns an event at 15:55. The record first becomes available to the strategy at 16:10. The proposed trade uses the 16:00 closing price. Even though the event occurred before the close, the information required by this signal arrived afterward.

Ledger entry	Time in the exercise	Consequence
Underlying event	15:55	Does not establish feed availability
Claimed closing fill	16:00	Precedes the usable signal
Signal input available	16:10	Earliest input boundary for this decision
Next eligible execution opportunity	Later, under the trading rules	Still requires a feasible order and fill

The repair is to move the decision to a time when the information exists, then model a permissible execution opportunity. That might be the next session under the stipulated strategy, but knowing the next session's opening price does not guarantee that your whole order fills there.

Preserve historical revisions too. If a restated value becomes available in 2026, do not use it as the original input to a 2020 decision. Retain the version known at that decision or explicitly narrow the experiment to data for which availability can be established. A chronological train/test split cannot repair future values already embedded in every row.

How does the historical universe affect returns?

Ask how instruments entered and left the eligible universe. Selecting today's surviving companies and pretending they were the full historical opportunity set can omit failures and delistings. The research guide above identifies survivorship bias; your audit must determine whether it affects this dataset.

Consider our equal-weight, single-period toy universe of three securities. A returns +10%, B returns 0%, and C returns -50%. With equal starting allocations, the portfolio return is approximately -13.3%. Retaining only surviving A would show +10%. The difference comes from sample construction, not improved signal quality.

This example deliberately holds weighting and the evaluation period fixed. A real audit also needs eligibility dates, corporate actions, trading suspensions, delisting proceeds, and the handling of unavailable prices. Do not infer these details from a clean price matrix that has already dropped problematic instruments.

Define exclusions before observing returns. If the strategy requires a minimum price or liquidity threshold, calculate eligibility using information available at the decision time. Today's membership in an index is not automatically the historical membership rule. Record the universe revision with the backtest so a later rerun does not silently evaluate a different opportunity set.

Is the fill price attainable?

QuantConnect's reality-modeling documentation separates fills, slippage, fees, brokerage rules, and other market constraints. It notes that default assumptions suit highly liquid assets and may need customization for illiquid instruments or large volume. Official reality-modeling concepts.

Our execution audit follows one order from decision to submission, acknowledgement, and fill. A signal calculated using the complete closing bar cannot casually receive that same close as an instantaneous fill. State the decision cut-off and order type, then check whether the corresponding venue mechanism admits the order in time.

Also distinguish an observed quote from executable volume. A small spread at one instant does not prove unlimited capacity. Partial fills, rejected orders, market hours, and position constraints can change the realized exposure. If the strategy depends on shorting, ask whether borrow availability and its costs were modeled.

A useful challenge changes order size while keeping the signal fixed. If the backtest assigns identical execution quality at every size, determine whether that is a defensible assumption or simply an unimplemented capacity model. This sensitivity check is our recommendation; it does not establish a universal market-impact formula.

How do costs turn gross edge into net results?

Define turnover before quoting transaction costs. In this original simplified model, turnover is the sum of absolute position changes expressed as a multiple of portfolio capital. A change from a -1 position to a +1 position trades two units of notional, not one. Opening and closing positions also count.

Suppose the period's gross return is 30 basis points, turnover is 4, and the total one-way cost is 8 basis points per unit of traded notional. The assumed drag is 4 × 8 = 32 basis points. Net return is therefore -2 basis points before any separately excluded financing or borrow costs.

Return bridge	Basis points
Gross strategy return	+30
Assumed turnover-scaled execution costs	-32
Net under these assumptions	-2

Specify what the 8-basis-point assumption includes. If spread and slippage are already represented by the simulated fill price, charging them again in a separate cost column double counts them. Conversely, commissions alone leave execution-price disadvantage unaccounted for. Reconcile the bridge against the trade ledger.

Then vary the uncertain inputs. With turnover fixed at 4, costs of 4, 8, and 12 basis points produce net returns of +14, -2, and -18 basis points. Also examine higher turnover during stressed periods. This exposes whether the result rests on a narrow execution assumption. It is more informative than attaching a single average fee to a strategy whose activity changes over time.

Why does repeated selection weaken evidence?

Bailey, Borwein, López de Prado, and Zhu study the probability of backtest overfitting and propose a framework with combinatorially symmetric cross-validation. Their central concern is selection among configurations that perform well in-sample but disappoint out-of-sample. Primary backtest-overfitting paper.

Our toy calculation illustrates multiple testing, not that paper's PBO estimator. If 100 independent null tests each have a 5% false-positive probability, the chance of at least one false positive is 1 - 0.95^100, approximately 99.4%. Independence is an explicit simplifying assumption. Correlated strategies require a different analysis; 99.4% is not an empirical probability that a particular strategy is overfit.

The practical implication is to preserve search history. Record parameter grids, feature alternatives, universe variants, cost settings, and discarded results. Showing only the winner removes information needed to assess how it was selected. A low-dimensional final model can still be the winner of a large hidden search.

If a researcher repeatedly checks the holdout and changes the strategy, that period becomes part of selection. Calling it “out-of-sample” afterward does not restore its independence from the choices made. Freeze the evaluation rule before using a final untouched period, and report what was changed if that boundary was crossed.

This article does not calculate PBO, a deflated Sharpe ratio, or a significance-adjusted investment result. Those require actual return series, research history, and stated statistical assumptions. The toy probability only demonstrates why one successful test cannot summarize a hundred attempted ones.

What should a defensible evaluation report contain?

Report the return definition, sampling frequency, exposure, costs, drawdown, and evaluation boundaries together. Our short equity path is 100 → 120 → 90 → 110. Final return is +10%, but the decline from the 120 peak to 90 is a 25% drawdown. The endpoint hides the path experienced by the investor.

Do not confuse that four-point illustration with an annualized result. Annualization requires a time horizon and assumptions about the sampled returns. A Sharpe calculation also needs an excess-return definition and attention to dependence. State the method before comparing two headline values.

Break results into relevant subperiods and exposure regimes without searching for a flattering presentation after the fact. Ask whether a few trades dominate the gain, whether performance changes with execution assumptions, and whether the strategy remains implementable at the proposed scale. A stress test should target a plausible failure mechanism, not merely add arbitrary noise.

For reproducibility, retain data and universe versions, availability rules, code revision, configuration, random seeds where relevant, and the full trade ledger. A saved plot is not enough to rerun a strategy. Keep the original research registry beside the final evaluation so reviewers can distinguish the frozen test from exploratory analysis.

How would you diagnose a suspicious curve aloud?

Start with one falsifiable concern. In the announcement exercise: “The signal arrives at 16:10, but the recorded fill is at 16:00. I would inspect the availability field and order timestamp before interpreting returns.” Then explain the repair and which parts of the result must be recomputed.

If fixing availability reduces performance, report the reduction rather than restoring the old assumption. Next audit the universe, then fills and costs, then the selection record. Keeping these changes separate makes their contributions visible. Changing every assumption simultaneously may produce a cleaner result while obscuring which defect mattered.

End with the limits of the evidence. A corrected historical simulation can support further evaluation under its assumptions. It does not guarantee future returns or replace execution validation. This is the conclusion of the audit, not a reason to avoid doing the audit carefully.

Practice backtesting interview questions

Question	Audit follow-up to rehearse
Design and Debug a Point-in-Time Backtest Data Pipeline	Reconstruct availability and historical data revisions.
Design and backtest a trading strategy	Present the hypothesis and the complete trial registry.
Model and measure trading transaction costs	Reconcile turnover, fill prices, and net returns.
Predict Stock Returns from Sentiment and Market Data	Establish when text and market features become usable.
t-Stat With an Added Orthogonal Regressor, Live Sharpe Check, and Absolute-Loss Estimator	Explain statistical assumptions behind a performance claim.

Choose another Quantitative Researcher practice question and bring an assumption ledger instead of only a chart. Your answer should identify the observation that could invalidate the claimed edge.

Sources and Further Reading

• QuantConnect: Research guide and bias hazards
• QuantConnect: Reality modeling, fills, slippage, and fees
• Bailey et al.: The Probability of Backtest Overfitting

Research checked October 4, 2026. Synthetic timestamp, turnover, multiple-testing, and drawdown calculations were checked locally. No live trading or historical strategy-performance experiment was run.