TradehouseBuyer guides & toolsPricingPart of TradeHouseTradeHouse Fidelity Lab · Buyer resources
Fidelity Lab vs doing your own walk-forward test
A walk-forward test you write yourself is a good habit and, for many strategies, the right first step. The question this page answers is narrower: what does a hand-rolled walk-forward usually leave out, and is closing those gaps worth $99 or a monthly plan? The honest answer depends on how many variants you tried before settling on the one you are about to trade.
What a typical do-it-yourself walk-forward looks like
Most home-built versions follow the same outline. You pick a training window, fit or tune parameters, test on the following window, roll forward and stitch the out-of-sample segments into one equity curve. That procedure is sound. Where it tends to break is in the details around it: the fee model is a flat number copied from a broker page, the threshold or rule was chosen after looking at the whole history, and nobody records how many other rules were tried on the way.
Where the gaps usually are
| Question | Typical DIY walk-forward | Fidelity Lab |
|---|---|---|
| Are fees real? | One flat fee or none, copied from a fee page | Ignores the fee in your file and re-nets every leg from a per-venue schedule; each leg is labelled SOT_MODELED, SOT_REALIZED or FLOOR_UNMODELED, and the last one applies a 2% floor and says so |
| Was the rule frozen before the test? | Depends on discipline; the test window is often peeked at | The threshold is frozen on the training segment and scored on held-out data it never touched |
| How many things did you try? | Rarely recorded | Benjamini-Hochberg at q=0.05 across every train-frozen rule you submitted, so twenty cohorts raise the bar |
| Is the sample big enough? | Counts rows (legs) | Counts day-clustered effective sample size with lag 1 to 5 autocorrelation inflation; independent days are the binding constraint, not legs |
| Is the data itself clean? | Assumed | 11 contamination detectors: lookahead, oracle lag, crossed book, signed-PnL direction, hindsight selection, mark tautology, unrealised counted as P&L, zero fee, duplicate legs, degenerate pin, span adequacy |
| Can it say no? | Always produces a curve | Returns SURVIVES, DISSOLVES, or UNTESTABLE when the book cannot answer the question |
The multiplicity problem is the one that costs money
Suppose you test 40 parameter combinations and keep the best. Even if none of them has any edge, the best of 40 will often look good out of sample by chance alone, and a plain walk-forward on that winner will report it faithfully. The test is correct; the selection is what contaminated it. A validator that sees all the cohorts you submitted can raise the significance bar accordingly. Doing that yourself requires a record of every variant and a correction such as Benjamini-Hochberg, which is simple to apply but easy to skip when you are excited about a result.
Where the do-it-yourself route is the better choice
- You have a single, pre-registered rule with no tuning. There is little selection to correct for.
- The market is continuous-price and you already measure implementation shortfall against your own arrival price.
- You enjoy building and auditing the harness, and the strategy is small enough that $99 is a meaningful share of expected profit.
Where an independent check pays for itself
- You tried many variants before choosing one, which is the normal case.
- The market is a binary prediction market, where fees are non-trivial and the hit rate near 0 or 1 can be structural rather than skill.
- You are about to size up. A refusal that says "independent days, not legs, is the binding constraint" costs $99 and can save a drawdown.
Pricing
Fidelity Lab bills separately from any data subscription and grades trades you upload. Annual billing on the monthly plans is two months free.
| Fidelity Lab option | Price | What it covers |
|---|---|---|
| Free | $0 | 1 report a month, up to 5,000 legs, all 11 contamination detectors, cost-netted P&L and a full walk-forward verdict |
| One audit | $99 one-time | A single report without a subscription |
| Five-audit pack | $399 one-time | Five reports, useful when you want to re-test after fixing what the first one flagged |
| Starter | $49 per month | 10 reports a month, up to 100,000 legs, CSV and JSON upload, Benjamini-Hochberg control across every cohort you submit |
| Pro | $149 per month | Unlimited reports, up to 2,000,000 legs per run, report API and webhook, scheduled re-runs of a saved book |
A reasonable workflow is to run your own walk-forward first, fix anything obvious, then submit the one book you intend to trade. If the verdict is DISSOLVES, the report states whether out-of-sample decay or real fees did the damage, which tells you whether to rework the signal or the execution.
Frequently asked questions
Does Fidelity Lab replace walk-forward testing?
No. It runs a point-in-time walk-forward as one part of the verdict and adds fee re-netting, contamination checks and multiplicity control around it.
Do I need to share my strategy code?
No. You upload trades (CSV or JSON). The report grades what happened in the file, not how the rule was written.
What if my data is too short?
The report returns UNTESTABLE and names the constraint, for example too few independent days, instead of inventing a grade.