TradehouseBuyer guides & toolsPricingPart of TradeHouseTradeHouse Fidelity Lab · Buyer resources
What is backtesting in machine learning?
In machine learning, backtesting means evaluating a model by replaying it on historical data in the order events actually happened, making each prediction using only what was known at that moment. It is the standard way to test forecasting models for sales, demand, prices and risk, and the only honest way to test a model that will be used on future data in a time series.
Backtesting vs ordinary cross-validation
Standard k-fold cross-validation shuffles rows and holds out random subsets. That works when rows are independent. In time series they are not, and shuffling lets the model see the future: a training row from Friday can sit next to a test row from Thursday. The score looks great and means little.
| Random k-fold | Backtest / walk-forward | |
|---|---|---|
| Split rule | Random rows | Strictly by time: train before, test after |
| Uses future data to predict the past? | Yes, often | No |
| Respects autocorrelation | No | Yes |
| Number of folds | Any | Limited by how much history you have |
| Suitable for time series | Rarely | Yes |
The basic procedure
- Order data by time and choose a first training window.
- Fit the model on that window only, including any scaling or feature selection.
- Predict the next block and record the error.
- Move the window forward and repeat, either expanding it or rolling a fixed length.
- Aggregate the errors across all test blocks. The spread across blocks is as informative as the mean.
Leakage: the most common backtest bug
Leakage means information from the test period reached the model through a side door. Typical routes: normalising with statistics computed on the full dataset, selecting features using all of the data, labels built from a future window that overlaps the training window, and joining a table that was updated after the prediction date. In finance, a related problem is using a revised data point that was not available at the time. A backtest should be point-in-time: every input as it stood on the prediction date.
Purging and embargo
When labels span several days, for example a five-day forward return, training samples near the boundary share information with test samples. Purging removes training samples whose labels overlap the test window, and an embargo leaves a gap after the test window before training resumes. Without them, a model can appear predictive simply because neighbours in time resemble each other.
Backtesting a model is not the same as validating a strategy
A prediction model can have low error and still lose money, because costs, spreads and position sizing sit between a forecast and a profit. And a strategy backtest adds a second layer of risk: the number of ideas tried. Evaluating a model on error metrics answers whether it forecasts; evaluating a strategy answers whether trading it pays after fees and selection.
How Fidelity Lab relates
Fidelity Lab works on the strategy side. You upload the trades your model or rules produced, and it checks for lookahead and other data artefacts, re-nets fees, runs a walk-forward with the threshold frozen on training data, and applies multiplicity control. The first report each month is free. It does not retrain your model and never sees your code.
A minimal checklist
- Every feature is computed from data available at the prediction timestamp.
- Scalers and encoders are fitted inside each training window.
- Labels that look forward are purged near the boundary.
- The same transaction costs will apply live as in the test.
- You report the spread of fold results, not only the average.
- You have recorded how many model variants were evaluated.
Frequently asked questions
Is backtesting the same as testing on a holdout set?
A single holdout is one backtest fold. Walk-forward backtesting repeats the idea over many consecutive periods.
Can I backtest a model with random splits?
Only when rows are genuinely independent. For prices, demand and similar series, use time-ordered splits.
What metric should I use?
For forecasts, error metrics such as MAE or RMSE per block. For trading, net P&L after real costs and a significance check that accounts for how many ideas you tried.