How to backtest a trading strategy without fooling yourself
A backtest can show almost anything you want it to. This is the checklist we use, with an example from our own 404 tested variants for every step.
Short answer: write the rules and the pass bar before you look at the data, charge realistic costs, split your history into development, validation and a holdout, count every variant you try, compare with a random-entry control matched to your signal, test a second market, and publish everything, including what failed.
01Write the rules before you look at the data
If you adjust a rule after seeing results, you are fitting the past, not testing an idea. Write the exact rules, the costs, the periods and the pass bar down, and date them. Every test here has a frozen, dated specification linked from its post; the forex study's specification was published before any forex result was run. See the forex specification.
02Use data that matches how you will trade, and know its limits
Our ES tests use real futures data. Our gold and forex tests use broker spot history, which records no spreads and may contain interpolated bars in early years. We say so on every post, because it limits what the results can claim.
03Charge realistic costs on every trade
Costs are the quiet killer of active strategies. On ES we charged about $31 per round trip. Across the 28 ES variants (41,226 trades) the strategies made +$1,264,564 before costs and -$3,548 after. The MACD crossover made +$4,088 before costs over 3,249 trades and -$95,852 after. Try your own numbers in the trading cost calculator.
04Split your history: development, validation, holdout
Build on the first period, check on the second, and only then look at the third. Say honestly if the holdout was already used. Of our 404 variants, 393 failed the development t-statistic gate, and the gold breakout that looked best in development (profit factor 1.41) fell to 0.86 in validation. See what is a good profit factor.
05Count every variant you try
Try enough versions and some will look good by luck. Report how many you tried, and make the bar stricter as the count grows: our development t-statistic bar rises from 2.0 to 2.5 when more than four variants of one strategy have been tried. Across all our tests we tried 404 variants and 2 passed every gate.
06Compare with a random-entry control, and match it to your signal
Run the same exits and costs on random entries and see where your result falls among 500 random runs. The control has to resemble the signal. Our first control for RSI divergence only bought below RSI 50, while about half the real signals fired above it: the long-only version ranked at the 78th percentile against that control and at the 99th against one matched on RSI at entry. For Bollinger, a distance-matched control moved the long-only target-and-stop version from the 99th percentile to the 90th.
07Test a second market
An edge that holds on several markets is more convincing, although an edge can exist in one market only. Bollinger's long side was our best hint on ES; on four currency pairs it finished above zero in 0 of its 16 results. See the forex study. Our first survivor, the 1-hour golden cross on gold, failed when the same rule was run on ES and was later made Inconclusive by a data check. Our two current survivors, the noise area breakout on NQ in the New York session, did not pass on ES or on gold, which is one reason we call them candidates and not results: read the NQ noise area test.
08Look at long and short separately, and at risk as an account feels it
On ES the short side lost money in 14 of the 14 long-and-short variants, and the long side's profits mostly tracked a rising market. Report drawdown including open losses, not only on closed trades.
09Publish everything and log your changes
Show every variant, not the best one, and keep a corrections log when you change a claim. All our posts have one, and the checker that compares each page with its results files is described in the protocol.
10Where to go next
What all the backtests found so far · Glossary of backtesting terms · Trading cost calculator · Every test by market.