Survivorship bias

To test a rule on 2014 honestly, you need the list of stocks that existed in 2014, including the ones that later went bankrupt, were bought out or shrank into irrelevance. We do not have that list. We have the list of companies we cover today.

Why it matters#

• Every company in today's list survived. A rule that picked a stock in 2014 could only pick one that is still here.
• Today's largest companies are in the pool from the first day, before anyone knew they would become the largest.
• Failures that would have cost the strategy real money are simply absent.

How large is the effect#

We cannot put a number on it for our own tests, because that would need the missing data. As a general matter it is worst for strategies that favour small, cheap or distressed companies, since those fail most often, and smaller for strategies that hold the largest companies.

What we do about it#

• Every stock backtest says how many companies were in the pool and that the pool is today's list.
• We tell you to read the lead over the S&P 500 as a ceiling, not an estimate.
• Fund-based strategies pick from a fixed list of index and sector funds. Funds do not go bankrupt the way companies do, so the problem is much smaller there, and the page says which kind of pool was used.
• The live record does not have this problem. It only counts what happened after the picks were made.

A related gap: company size in the past

We do not hold historical share counts. Where a rule ranks by size, past size is today's share count times the price back then. Buybacks and new share issues make the older rankings rough, and the backtest says so.