Scams, myths and investor protectionModule 2, lesson 6 of 15
Misleading backtests and cherry-picked results
Why a strategy's impressive past test results can be an illusion, from cherry-picked dates and curve-fitting to ignored costs, and what to ask before trusting one.
In this lesson
- Explain what a backtest is and what it can and cannot show.
- Explain how testing many variations produces a winner by chance.
- Recognise cherry-picked periods, ignored costs and look-ahead errors in performance claims.
- Ask the questions that separate a credible track record from a sales pitch.
Builds on
- Fake testimonials, screenshots and misleading advertisementsBacktests are another kind of selected evidence, used in the same adverts.
- Misleading numbers and chartsCherry-picked start and end dates, covered there, are one of the main ways backtests mislead.
On this page
In simple terms
A backtest takes a set of trading rules, such as "buy when the price rises above its 50-day average, sell when it falls below", and works out how those rules would have performed on past prices. It is a useful research tool. It is also one of the easiest kinds of evidence to make look impressive.
The reason is simple: the past is fixed and fully known. With enough attempts, you can always find rules that would have done brilliantly on it. That does not mean those rules know anything about the future.
Trying many rules finds a lucky winner
Suppose you test a rule with dozens of settings: different averages, different holding periods, different shares. Each version has some result, and by chance some will be far better than others. If you then pick the best and show only that one, you are showing luck as if it were skill.
Worked example: 200 versions of a rule
A made-up seller tests 200 versions of a trading rule on five years of past prices. Suppose, for this example, that none of the versions has any real edge: their results are spread around zero after costs, like the results of tossing coins, and by chance about 1 version in 40 beats 20% a year in the test.
| Versions tested | 200 |
|---|---|
| Versions that happen to beat 20% a year: 200 ÷ 40 | about 5 |
| Version shown in the advert | the single best one |
| Expected real edge of that version | none |
The advert says "our strategy returned 41% a year over five years". Every number is real arithmetic on real past prices. But the strategy was chosen because it fitted the past best, and its future results are likely to look like those of the other 199: around zero after costs.
Fitting rules so closely to past data that they capture random noise rather than a lasting pattern is called overfitting, or curve-fitting. Warning signs include many conditions in the rules ("buy on Tuesdays when the 13-day average crosses the 47-day average and volume is above...") and settings that look oddly precise.
Other ways a backtest misleads
Cherry-picked periods. Starting a test just after a crash, or ending it just before one, can transform the result. The misleading statistics lesson shows how much the choice of dates matters.
Ignored costs. Brokerage, taxes, the bid-ask spread and slippage are paid on every trade. A strategy that trades often can look profitable before costs and lose money after them.
Worked example: What costs do to a busy strategy
Suppose a strategy makes 250 round trips a year, and each round trip costs 0.2% of the money traded in charges and spread. This is a made-up cost figure for the example.
| Cost per round trip | 0.2% |
|---|---|
| Round trips a year | 250 |
| Cost a year, roughly: 250 × 0.2% | 50% of the money traded |
A backtest that ignores costs could show a 30% yearly gain for a strategy that loses heavily in practice.
Survivors only. Testing on today's index members, or on companies still listed, leaves out the companies that failed or were removed along the way. That makes almost any buying strategy look better than it would have been.
Look-ahead errors. A test can accidentally use information that was not available at the time, such as annual results that were published months after the year they cover, or a closing price used to decide a trade placed earlier that same day.
Unrealistic trading. Assuming every order fills at the exact price shown, even in thinly traded shares or during a sharp fall, ignores the practical limits described in the stop-loss orders lesson.
What better evidence looks like
A credible performance claim has these features:
- Tested on data the rules were not built on. Rules developed on one period, then tested once on a later period, without changes in between.
- Traded live with real money, not just simulated, with every trade recorded.
- After all costs and taxes.
- Over a long period that includes rising, falling and sideways markets.
- Checked by someone independent of the seller.
- Honest about the bad times: the largest fall from a peak and the longest losing run, not only the average.
Sellers of trading systems and "algos" rarely offer this. A backtest alone, especially one with a smooth rising line and no losing years, is a sales tool.
Why it matters
Backtested returns are used to sell courses, trading software, signal services and "automated strategies". They look scientific, with charts and precise percentages, which makes them more persuasive than a simple testimonial. Knowing how easily they are produced lets you treat them as what they are: an untested idea.
Limitations
Backtesting is not worthless. Researchers use it carefully to reject bad ideas and to understand how a rule behaves in different conditions. The problem is not testing on past data; it is selecting the best of many tests and presenting it as proof. Even a well-designed test on fresh data can fail in future, because markets change.
Common mistakes
- Treating "would have made" as "made". A simulation is not a track record.
- Being impressed by precision. "41.7% a year" is no more reliable than "about 40%".
- Ignoring how many ideas were tried. The more were tested, the less the winner means.
- Forgetting costs. Frequent trading strategies are the most sensitive to them.
- Trusting a smooth line. Real strategies have losing periods. A curve with none is a warning sign.
Try it yourself
Find an advert or video promoting a trading strategy or "algo" with past results. Without paying for anything, write down:
- Is the result a backtest or a record of real trades?
- Which period does it cover, and does it include a major market fall?
- Are costs, taxes and slippage included?
- How many versions of the strategy were tried before this one was chosen? If it does not say, assume many.
- Who checked the result?
Count how many of the five you could answer from what the seller provides.
Key takeaways
- A backtest shows how a set of rules would have done on past prices. It is a hypothesis about the past, not a record of real trading.
- Test enough variations of a strategy and one will look excellent by luck. The more rules were tried, the less the winner's result means.
- Backtests often leave out trading costs, slippage and taxes, use only today's surviving companies, or accidentally use information that was not available at the time.
- The best evidence is a strategy tested on data it was not built on, then traded with real money, with every trade recorded, over a long period.
- A seller who shows only a backtest, and no live record, is asking you to trust a simulation.
Check your understanding
Finished reading?
Your progress is saved only in this browser.
Terms used in this lesson
- Backtest
- A test of how a set of trading or investing rules would have performed on past prices, as if they had been followed at the time.
- Overfitting
- Tuning rules so closely to past data that they capture random noise rather than a lasting pattern, so they look excellent in tests and fail afterwards.
- Cherry-picking
- Choosing the data, dates or examples that support a claim while leaving out the ones that do not.
- Survivorship bias
- The error of judging something by the examples that survived or succeeded, while ignoring the many that failed and dropped out of view.
References
- Due diligence, SEBI investor website. SEBI's guidance on checks to make before investing.
Spotted a mistake or something unclear? Report it by email. Here is how corrections work.
This lesson is general education, not investment advice. Examples use made-up or clearly dated figures to explain ideas; they are not recommendations. Read the disclaimer.