Survivorship, look-ahead and other backtesting biases

In simple terms

A backtest uses real historical prices, so it is tempting to assume its results are realistic. They often are not. Several well-known biases make backtests look better than any trader could have done in reality. The misleading backtests lesson introduced them from an investor's point of view; this lesson looks at them from the side of someone building a test.

Survivorship bias

Survivorship bias arises when a test uses only shares that exist today, or that are in an index today. Companies that went bankrupt, were delisted or fell out of the index are missing. A trader at the time would have bought some of them.

The fix is point-in-time data: the list of shares that were actually available, or in the index, on each past date, including those since delisted.

Look-ahead bias

Look-ahead bias arises when the test uses information that was not available when the trade would have been made. It creeps in through:

• Same-bar execution: generating a signal from today's close and assuming you bought at that same close. You could only have bought at the next available price.
• Fundamental data: using a year's results on the first day of the next year, before they were published, as the misleading backtests lesson explains.
• Revised data: using figures that were later restated.
• Index membership: choosing shares by today's index, which is itself information from the future.
• Parameter choices: choosing settings by looking at the whole period's results, discussed in the overfitting lesson.

Data problems

• Unadjusted prices: splits and bonus issues appear as crashes, triggering false signals, as the adjusted prices lesson explains.
• Missing dividends: strategies that hold shares miss part of their return; strategies that sell before ex-dates may look better than they were.
• Bad ticks: erroneous prices in the data, such as a single trade at a wildly wrong price, can create profits that never existed.

Unrealistic execution

• Fills at the exact signal price ignore the spread and slippage.
• Unlimited size: assuming you could buy any quantity at the quoted price, even in thinly traded shares.
• Gaps: assuming stops fill at their trigger, when gaps would have filled them much worse, as the price gaps lesson shows.
• Price bands and circuit limits: assuming you could sell a share locked at its lower circuit, when in reality there were no buyers.

Ignored costs

Brokerage, taxes, exchange charges, the spread and slippage are paid on every trade, as the backtesting lesson explains. Leaving them out is the simplest way to make a weak strategy look strong.

A checklist for any backtest

Why it matters

Each bias pushes results in the same direction: better than reality. Together they can turn a losing idea into an apparently excellent one. Knowing them lets you build fairer tests yourself and see through the impressive backtests used to sell courses and strategies.

Limitations

Some biases are hard to remove completely: point-in-time data for delisted shares can be hard to obtain, and slippage can only be estimated. Even a bias-free backtest describes the past, not the future.

Common mistakes

• Testing on today's index members.
• Trading at the close that generated the signal.
• Using unadjusted data.
• Assuming any size fills at the quoted price.
• Leaving out costs.

Try it yourself

1. Take a backtest result you have seen, in a course, video or article.
2. Work through the checklist and note which questions the source answers.
3. For each unanswered question, write down which way the bias would move the result.
4. Decide how much of the claimed result you would still believe.