Technical analysisModule 6, lesson 31 of 32
Overfitting, sample size and out-of-sample testing
How tuning a strategy's rules fits random noise, why small samples mislead, how out-of-sample and walk-forward tests work, and the signs of a robust result.
In this lesson
- Explain how adding rules and tuning settings leads to overfitting.
- Explain why small samples of trades give unreliable results.
- Describe out-of-sample and walk-forward testing.
- List the signs that a backtested result is robust rather than fitted.
Builds on
- Survivorship, look-ahead and other backtesting biasesOverfitting is the bias that comes from how the rules themselves are chosen.
- Correlation and causationFitted patterns are correlations in past data that may have no lasting cause.
On this page
In simple terms
Given enough freedom, you can always find rules that would have made money in the past. Add a filter here, adjust a setting there, and the backtest gets better and better. But each adjustment makes the rules fit the particular wiggles of past prices more closely, and those wiggles were mostly random. This is overfitting, also called curve-fitting.
An overfitted strategy looks excellent on the data used to build it and fails on new data. The misleading backtests lesson showed this from an investor's point of view; this lesson shows how to guard against it when testing yourself.
How overfitting happens
Many versions tested. Testing 100 settings and choosing the best is like tossing 100 coins and choosing the one that landed heads most often: you have selected luck.
Many rules. Each extra condition ("only on Tuesdays", "only when volume is between 1.3 and 1.8 times average") can remove a few bad trades from the past. It almost always improves the backtest and almost never improves the future.
Precise settings. A strategy that only works with a 47-day average, and not with a 45- or 50-day one, is probably fitted to noise.
Peeking. Looking at the whole period's results, then adjusting the rules, then testing again on the same period, overfits even without meaning to. The correlation and causation lesson explains why patterns found this way often have no lasting cause.
Small samples
A backtest's result rests on the trades it contains. With few trades, luck dominates.
Worked example: How much can luck move a small sample?
Suppose a strategy has no real edge: each trade is equally likely to win or lose the same amount.
| Chance of 18 or more wins in 30 trades (a 60% win rate or better) | about 18%, roughly 1 in 5 |
|---|---|
| Chance of 13 or more wins in 20 trades (65% or better) | about 13%, roughly 1 in 8 |
| Chance of 60 or more wins in 100 trades (60% or better) | about 3% |
With 30 trades, roughly one useless strategy in five shows a win rate of 60% or more by luck. With 100 trades, that becomes much rarer. A trader who sees a 60% win rate on 30 trades may believe they have found something when they have found nothing.
The more trades a test includes, across different market conditions, the more its result reflects the rules rather than luck. Even then, a long test covers only the conditions that happened in that period.
Out-of-sample testing
The core defence is to keep some data untouched:
- Split the data. For example, use the first seven years to develop the rules (the in-sample period) and keep the last three years aside (the out-of-sample period).
- Develop the rules using only the in-sample data.
- Test once on the out-of-sample data, without further changes.
- Accept the out-of-sample result, good or bad. If you change the rules after seeing it and test again, it is no longer out of sample.
Walk-forward testing
Walk-forward testing repeats the process through time: build on years 1 to 5, test on year 6; build on years 2 to 6, test on year 7; and so on. The combined out-of-sample results give a more realistic picture of how a method that is periodically re-tuned would have done.
Worked example: A made-up walk-forward summary
| Test year | In-sample return of chosen settings | Out-of-sample return |
|---|---|---|
| Year 6 | 28% | 6% |
| Year 7 | 31% | −3% |
| Year 8 | 26% | 2% |
| Year 9 | 33% | −7% |
The in-sample returns are consistently high and the out-of-sample returns hover around zero. This is the classic signature of overfitting: the tuning finds patterns that do not persist.
Signs of a robust result
- Few rules and settings, with a plausible reason why the effect might exist.
- Stable across nearby settings: a 45-, 50- and 55-day average all give similar results.
- Works in different periods and different shares, not just one.
- Holds up out of sample and in walk-forward tests.
- Survives costs, with room to spare.
- Many trades, across rising, falling and sideways markets.
Why it matters
Overfitting is the single most common reason backtested strategies fail in real trading. Understanding it changes how you read every performance claim and how you test your own ideas: fewer rules, more data, fresh data for the final test, and acceptance of disappointing answers.
Limitations
Out-of-sample tests reduce overfitting but do not eliminate it: if you try many ideas and keep only those that pass out of sample, you can overfit to the out-of-sample period too. Markets change, so even a robust past result can fade. No test can guarantee future performance.
Common mistakes
- Choosing the best of many settings and believing its result.
- Adding rules until the backtest looks good.
- Re-testing on the out-of-sample data after changing the rules.
- Trusting results from a few dozen trades.
Try it yourself
- Take a simple rule, such as a moving-average crossover, and five years of adjusted data for one share.
- Use the first three years to choose the best average length from a few options.
- Test that length once on the last two years.
- Compare the two results, and repeat with nearby lengths to see how stable they are.
Key takeaways
- Overfitting happens when rules and settings are tuned until they fit past noise. The result looks excellent on the data used to build it and fails afterwards.
- Every extra rule, filter or tuned setting adds a chance to fit noise. Simple strategies with few settings are more likely to be real.
- Small samples of trades are dominated by luck. Dozens of trades can show a strong result by chance.
- Out-of-sample testing keeps part of the data untouched while building the strategy, then tests it once on that data. Walk-forward testing repeats this through time.
- A robust result works across nearby settings, different periods and different shares, with a plausible reason why it should work.
Check your understanding
Finished reading?
Your progress is saved only in this browser.
Terms used in this lesson
- Overfitting
- Tuning rules so closely to past data that they capture random noise rather than a lasting pattern, so they look excellent in tests and fail afterwards.
- Out-of-sample testing
- Testing a strategy once on data that was kept aside and not used while building it, to see whether its results hold on new data.
- Backtest
- A test of how a set of trading or investing rules would have performed on past prices, as if they had been followed at the time.
Spotted a mistake or something unclear? Report it by email. Here is how corrections work.
This lesson is general education, not investment advice. Examples use made-up or clearly dated figures to explain ideas; they are not recommendations. Read the disclaimer.