A common pitfall of examining historical data for clues of predictive value is hindsight bias.
Hindsight bias occurs when a person believes the outcome of an event was predictable in the past. In the world of investing, we commonly see this when someone claims they knew they should have bought a certain stock while it was in the early stages of a big run or when they say they knew it was time to get out of the market before a big downturn occurred.
Hindsight bias is related to the human tendency to see patterns in how events play out. Early man learned to identify various patterns as part of his survival instinct. Learning the migration habits of animals helped to hunt. Similarly, learning seasonal weather patterns allowed our ancestors to transition from being nomads to becoming farmers.
The problem with our tendency to recognize patterns is its effect on how we view both random events and events whose outcomes were not so obvious at some prior point. We naturally default to assuming a known data point, statistic or outcome can be used to analyze historical data. This assumption ignores the possibility of the data point or statistic evolving over time. It also ignores the possibility of a different outcome having occurred if the data was viewed differently at some point in the past.
A way to avoid hindsight bias is to examine data as it existed in the past. In other words, the analysis must only be limited to the information that was available to the individual at a certain point of time. In order to do this, the data must also be clear of survivorship bias. Survivorship bias occurs when components of the data universe are dropped over time. A mutual fund database would have survivorship bias if it excluded all funds that have been closed or merged into other funds at some point in the past. An analysis of a trading strategy would have survivorship bias if it excluded stocks that no longer exist. These are stocks that not only existed in the past, but that also would have been identified by the strategy on a previous date. A study without hindsight bias and survivorship bias measures what would have happened if the decision was made based on the information available to a person at a given point in time.
To show how hindsight bias can affect the outcome of analysis, I conducted two new analyses of the AAII Sentiment Survey. The first analysis determined what was an unusually high or low reading based on the historical data available on a given past date in time. The second used the measures of unusually high and low readings as we currently define them to provide a comparison. The AAII Sentiment Survey is a useful data group to use since we have been logging the weekly results ever since the poll was first conducted in July 1987.
There is also a second benefit to this analysis. Investors can see how the S&P 500 index performed when aggregate sentiment about the short-term direction of stock prices was positive (bullish), neutral or negative (bearish). In other words, this analysis looks at the historical synchrony between the AAII Sentiment Survey and the performance of the market.