Explainer · Charts & Analysis
Backtesting pitfalls: why past results mislead
A backtest runs a trading rule over old prices to see how it would have done. It is a useful way to reject bad ideas, and a dangerous way to fall in love with them.
Quick answer
Backtesting means applying a strategy to past market data. The results are hypothetical: U.S. rules class backtested results as hypothetical performance [2], and the NFA's required disclaimer says hypothetical and actual results frequently differ sharply [3]. Treat a good backtest as a reason to test further, not to trade bigger.
Key points
- A backtest is a simulation built with hindsight; it is not trading results [4].
- Testing many variations and keeping the best one finds luck as easily as skill; in our simulation of 200 random rules, the best gained 56.4% and then lost 15.8% the next year (calculated).
- Data sets that only contain survivors look better than reality, because the losers dropped out [6].
- Backtests cannot fully account for illiquidity and slippage, and they involve no real money at risk [4].
- Rules that once worked can stop working: simple currency trading rules lost their returns after the 1970s and 80s [5].
On this page
What is backtesting?#
A backtest is a simulation of how a set of trading rules would have performed on historical market data. Researchers note that many investment firms rely on backtests to choose strategies and allocate capital [1].
The key word is would. U.S. securities rules list performance that is backtested, by applying a strategy to data from prior periods when it was not actually used, as a form of hypothetical performance [2]. The National Futures Association (NFA) rulebook sets out a disclaimer for hypothetical results stating that no representation is made that any account will or is likely to achieve similar profits or losses [3]. That is the honest starting point for reading any backtest, including your own.
Why do backtests look better than live trading?#
Because they are built after the fact. The NFA's disclaimer says hypothetical results are generally prepared with the benefit of hindsight, and that there are frequently sharp differences between hypothetical results and the actual results later achieved [3]. Its guidance adds two more reasons: simulated trading cannot fully account for a lack of liquidity and price slippage, and because no real money is at risk, it cannot completely account for the impact of factors associated with risk [4].
The flow below shows where the gap opens up. Each step looks harmless on its own.
What is overfitting, and why is it so easy?#
Overfitting means tuning a rule so closely to past data that it captures noise rather than anything that will repeat. The St. Louis Fed warns that apparent trading-rule profits can be spurious because of data snooping, publication bias and data mining [5]. Even standard safeguards can fail: researchers on backtest overfitting found that the usual hold-out method tends to be unreliable and inaccurate for investment backtests [1].
To see how easily luck looks like skill, we simulated it. We created 200 rules that have no edge at all: each day, each rule gains 1% or loses 1% on a coin flip. We ran them for a year of 250 trading days, picked the winners, then ran the same rules for another year.
| Group of rules | Year one (the backtest) | Year two (same rules) |
|---|---|---|
| Single best rule | +56.4% | -15.8% |
| Average of the top 10 | +44.3% | -6.2% |
| Average of all 200 | +0.9% | -0.1% |
Simulated with random numbers in Python (seed 42), 200 coin-flip rules, 250 days each year, compounded. All figures calculated; no real market data.
In year one, 33 of the 200 random rules gained more than 20% (calculated). None of them knew anything. If you test enough settings of a moving average or a candlestick pattern, some will look brilliant by chance, and the one you pick may simply be the luckiest.
What is survivorship bias?#
Survivorship bias happens when your data only includes the things that are still around. A classic study of mutual funds explains the problem: funds that disappear tend to do so because of poor performance [6]. Leave them out and the average looks better than what investors actually experienced.
The same trap applies to a backtest on today's list of stocks or coins. The ones that collapsed, merged or were delisted may simply be missing from the data you download.
How much can costs and slippage change a backtest?#
Often the whole result. A study of candlestick strategies on Dow Jones stocks found that the same reversal patterns were profitable at a 0.5% transaction cost under one holding rule and not profitable under another [7]. Small assumptions about how and when you exit decide the outcome. Costs also differ by who is trading: a St. Louis Fed paper cites spreads of 2 basis points or less since 2000 for currency trades of $5 million to $50 million [5]. That figure is for large institutional trades and tells you nothing about what a small account pays, so never borrow it for your own backtest.
Take an invented backtest of 100 trades a year with an average gross gain of 0.20% per trade.
- No costs assumed
- +20%100 x 0.20%, simple sum, calculated
- With 0.10% cost per trade
- +10%100 x (0.20% minus 0.10%), calculated
- Plus 0.05% slippage per trade
- +5%100 x (0.20% minus 0.10% minus 0.05%), calculated
- Share of the gross result left
- 25%5 / 20, calculated
Our trading cost calculator shows what your own broker's spread and fees do to a trade, and the slippage entry explains why fills drift from the price you planned.
How do regulators treat backtested results?#
Carefully. Under the SEC's marketing rule, advisers who show hypothetical performance must have policies to make it relevant to the audience and must explain the criteria and assumptions used, along with the risks and limitations of relying on it [2]. In 2023 the SEC charged nine advisers for advertising hypothetical performance to mass audiences on their websites without the required policies, and the firms agreed to pay $850,000 in combined penalties without admitting or denying the findings [8]. The charges were about missing policies, not proof that the backtests were false [8].
The NFA goes further for its members: hypothetical results are not allowed for a trading program once the member has three months of actual results, subject to an exception in its rule [4]. Those rules cover registered firms, not anonymous signal sellers online. When a stranger shows you a backtest with no live record, the protections above do not apply.
How can you test an idea more honestly?#
- Write the rules before you look
Entry, exit, stop and size, in plain words. Changing them after seeing results is how hindsight gets in [3].
- Count how many versions you tried
The more settings you test, the more likely the winner is luck [5]. Note every version, not just the best.
- Charge realistic costs
Use your own broker's spread and fees, then add slippage. If the edge disappears, so does the strategy.
- Check the data for missing losers
Ask whether delisted stocks or dead coins are included [6].
- Track it forward with tiny or no money
A forward record in a trading journal is closer to reality than any backtest, and size any real trades with position sizing.
Mistakes beginners make with backtesting#
- Tuning until it looks perfect
A smooth equity curve after dozens of tweaks can simply mean the rule learned the noise. Our simulation found a 56.4% winner among 200 random rules (calculated).
- Testing on today's winners
A list of assets that survived to today leaves out the ones that failed [6].
- Assuming perfect fills
Backtests cannot fully account for illiquidity and slippage [4]. Real stops and market orders fill at what is available.
- Believing a screenshot
Hypothetical results shown without a live record carry the NFA's warning for a reason: sharp differences from actual results are frequent [3].
- Raising size because the backtest was good
Confidence from a simulation is still confidence. Our page on overconfidence explains why that is costly.
Frequently asked questions#
Is backtesting useless?
No. It is good at rejecting ideas that would have failed even with hindsight. It is weak at proving an idea will work, because results are hypothetical and frequently differ sharply from actual trading [3].
How many years of data do I need for a backtest?
None of the sources we cite gives a minimum. More data helps only if the rule was fixed in advance; long histories can include periods when markets behaved differently, as the fading of old currency rules shows [5].
Is paper trading the same as a backtest?
Not quite. Paper trading happens forward in time, so you cannot use hindsight. Like a backtest, though, it involves no real money, and the NFA notes hypothetical trading cannot fully capture the effect of financial risk [4].
The bottom line#
A backtest answers a narrow question: how a fixed set of rules would have done on old data, with hindsight, without real risk and, unless you add them, without real costs. Use it to throw out weak ideas, count how many versions you tried, charge honest costs and then watch the idea forward with little or no money before trusting it. Read the risk disclosure before trading.
Sources
- The probability of backtest overfitting (abstract, Journal of Computational Finance).
- 17 CFR 275.206(4)-1 -- Investment adviser marketing" (eCFR).
- RULE 2-29. COMMUNICATIONS WITH THE PUBLIC AND PROMOTIONAL MATERIAL" (NFA Rulebook).
- 9025 - COMPLIANCE RULE 2-29: USE OF PROMOTIONAL MATERIAL CONTAINING HYPOTHETICAL PERFORMANCE RESULTS.
- Technical Analysis in the Foreign Exchange Market" (Federal Reserve Bank of St. Louis Working Paper 2011-001B).
- Survivorship Bias and Mutual Fund Performance (abstract, The Review of Financial Studies 9(4), 1996, pp. 1097-1120).
- Trend definition or holding strategy: What determines the profitability of candlestick charting?" (abstract, Journal of Banking & Finance 61(C), 2015, pp. 172-183).
- SEC Sweep Into Marketing Rule Violations Results in Charges Against Nine Investment Advisers" (Press Release 2023-173).
Education only. This page is not investment, tax or legal advice. Trading and crypto can lose you money. See our risk disclosure.
