TradehouseBuyer guides & toolsPricingPart of TradeHouseTradeHouse Fidelity Lab · Buyer resources
What is backtest overfitting?
Backtest overfitting is what happens when a trading rule has been shaped, deliberately or by accident, to fit the quirks of one historical sample rather than a pattern that repeats. The backtest looks excellent because the rule was chosen for exactly that reason. Out of sample, the quirks are absent and the performance disappears.
How it happens in practice
Almost nobody sets out to overfit. It accumulates through ordinary workflow:
- Parameter search. You test a lookback of 10, 20, 30 and 50 days and keep the best. Each test is another draw from a lottery, and only the winner is reported.
- Rule adjustment after a bad stretch. A drawdown in March leads to an exit filter that would have avoided March. The filter now carries information from the test period.
- Cohort selection. You analyse the traders, tickers or markets that were already profitable. This is hindsight selection and it makes the in-sample rank look predictive.
- Data artefacts. Stale quotes scored as fills, duplicate rows, or exit marks equal to the entry-time mid. The P&L is generated by the data, not the idea.
Warning signs
| Sign | Why it matters |
|---|---|
| Smooth equity curve with an unusually high Sharpe | Real edges are noisy. A very smooth curve often means the data or the rule contains the answer. |
| Performance collapses with a small parameter change | A robust idea has a plateau of good parameters, not one sharp peak. |
| Many variants tried, one reported | The reported result is the maximum of a distribution, not a typical draw. |
| Strong in-sample, flat or negative walk-forward | The edge did not generalise to data the rule had not seen. |
| Edge disappears once real fees are applied | Thin edges on binary markets can be fully absorbed by the fee schedule. |
| All profit comes from a few days | The effective independent sample is tiny even if the leg count is huge. |
Checks that catch it
- Walk-forward with a frozen rule. Fix the rule or threshold on a training window and score it on later data it never saw.
- Multiplicity control. Count every variant and apply a correction such as Benjamini-Hochberg at a stated false-discovery rate.
- Deflated Sharpe. Adjust the ratio for the number of trials, sample length, skew and kurtosis. The calculator shows how fast confidence falls as trials rise.
- Cost netting from a real fee schedule, not a guess.
- Data hygiene checks for lookahead, duplicates and mark tautologies.
How Fidelity Lab approaches it
Fidelity Lab takes a trade file and runs 11 contamination detectors, re-nets costs from a per-venue fee schedule, performs a point-in-time walk-forward with the threshold frozen on the training split, checks day-clustered effective sample size and applies Benjamini-Hochberg at q=0.05. The answer is one of three: SURVIVES, DISSOLVES (with the cause, out-of-sample decay or fees) or UNTESTABLE when the book cannot support a conclusion. The free tier covers one report a month up to 5,000 legs.
A note on certainty
None of these checks proves an edge exists. They remove the most common false positives. A strategy that survives them has earned a small live allocation, not full confidence.
A quick self-test before you trade
Write down, before looking at any new data, how many variants you tried, what the best one scored, and what score you would need to see on unseen data to keep going. Then collect a few more weeks of forward data, or hold back the most recent segment, and compare. If you cannot state the number of variants, assume it was larger than you think. People reliably undercount the exploratory runs they discarded, and those discarded runs are exactly what the multiplicity correction needs to know about.
Frequently asked questions
Is overfitting the same as curve fitting?
In trading the terms are used interchangeably. Both mean tuning a model to the noise of one sample.
Can a simple strategy be overfit?
Yes. Simplicity limits parameters but not the number of ideas you tried before settling on one.
How much out-of-sample data is enough?
It depends on the number of independent days, not rows. A year of daily data is roughly 250 independent observations at best.