Backtesting glossary

Plain-English definitions of the terms used across our backtests, with a link to where each one matters.

Backtest: A simulation of how a set of trading rules would have performed on historical data. It shows what the rules did in the past, not what they will do.
Gross and net result: Gross is the result before costs; net is after commissions, spread and slippage. A strategy can be profitable gross and a loser net. Costs in a backtest.
Profit factor: Gross profit divided by gross loss. Above 1.0 the strategy made money, below it lost. It ignores trade count and drawdown. What is a good profit factor?.
Average net per trade (expectancy): The average result of one trade after costs. It has to be larger than the cost per trade to matter, and it should come with an error range.
Costs and slippage: Commission, spread and the difference between the price you expected and the price you got. We charge them on every fill. Trading cost calculator.
Development, validation and holdout: Three consecutive periods of history. You build on the first, check on the second and look at the third last. A holdout that was used in earlier tests is flagged as already used. How we split the data.
Out-of-sample: Data the rules were not built or tuned on. Results that vanish out of sample were probably fitted to noise.
t-statistic: The average result per trade divided by its standard error. A larger value means the average is less likely to be luck. Our bar is 2.0, or 2.5 when more than four variants were tried.
Overfitting (curve fitting): Tuning rules until they fit past data, including its random wiggles, so the backtest looks great and the live result does not.
Multiple comparisons (counting your tries): The more variants you test, the more likely one looks good by luck. Reporting how many you tried, and raising the bar as the count grows, is the fix. All our variants.
Random-entry control: The same exits, costs and trade count applied to random entries, repeated 500 times. Your strategy has to beat the 95th percentile of these runs to count. The control in our protocol.
Matched control: A random control built to resemble the signal, for example entering only in the same state (trend, RSI level, distance from a band). Unmatched controls can make a signal look better or worse than it is. Example: RSI divergence.
Drawdown (closed and open): The fall from a peak in the account. A closed-trade drawdown ignores losses inside open trades; including them is the more honest measure of risk.
Market drift: The market's tendency to rise (or fall) over the test. A rising market makes long-only strategies look good whether or not the signal works. Example: Supertrend long only.
Frozen specification: The rules, costs, periods, gates and variants written down and dated before the data is run, so results cannot quietly change the question. Example: the forex specification.
R-multiple: A result expressed in units of the initial risk (R). We use it for gold, where the stop distance changes every day. Gold breakout post.
Pip: The standard small price step in forex: 0.0001 for most pairs and 0.01 for yen pairs. We report forex results in pips. Forex study.

How to backtest a trading strategy without fooling yourself · The protocol · What all the backtests found so far.