Close

Analyzing the AAII Sentiment Survey Without Hindsight

by Charles Rotblut, CFA

A common pitfall of examining historical data for clues of predictive value is hindsight bias.

Hindsight bias occurs when a person believes the outcome of an event was predictable in the past. In the world of investing, we commonly see this when someone claims they knew they should have bought a certain stock while it was in the early stages of a big run or when they say they knew it was time to get out of the market before a big downturn occurred.

Hindsight bias is related to the human tendency to see patterns in how events play out. Early man learned to identify various patterns as part of his survival instinct. Learning the migration habits of animals helped to hunt. Similarly, learning seasonal weather patterns allowed our ancestors to transition from being nomads to becoming farmers.

The problem with our tendency to recognize patterns is its effect on how we view both random events and events whose outcomes were not so obvious at some prior point. We naturally default to assuming a known data point, statistic or outcome can be used to analyze historical data. This assumption ignores the possibility of the data point or statistic evolving over time. It also ignores the possibility of a different outcome having occurred if the data was viewed differently at some point in the past.

A way to avoid hindsight bias is to examine data as it existed in the past. In other words, the analysis must only be limited to the information that was available to the individual at a certain point of time. In order to do this, the data must also be clear of survivorship bias. Survivorship bias occurs when components of the data universe are dropped over time. A mutual fund database would have survivorship bias if it excluded all funds that have been closed or merged into other funds at some point in the past. An analysis of a trading strategy would have survivorship bias if it excluded stocks that no longer exist. These are stocks that not only existed in the past, but that also would have been identified by the strategy on a previous date. A study without hindsight bias and survivorship bias measures what would have happened if the decision was made based on the information available to a person at a given point in time.

To show how hindsight bias can affect the outcome of analysis, I conducted two new analyses of the AAII Sentiment Survey. The first analysis determined what was an unusually high or low reading based on the historical data available on a given past date in time. The second used the measures of unusually high and low readings as we currently define them to provide a comparison. The AAII Sentiment Survey is a useful data group to use since we have been logging the weekly results ever since the poll was first conducted in July 1987.

There is also a second benefit to this analysis. Investors can see how the S&P 500 index performed when aggregate sentiment about the short-term direction of stock prices was positive (bullish), neutral or negative (bearish). In other words, this analysis looks at the historical synchrony between the AAII Sentiment Survey and the performance of the market.

The AAII Sentiment Survey

The AAII Sentiment Survey asks a simple question: “I feel that the direction of the stock market over the next six months will be:

  • Up—Bullish
  • No Change—Neutral
  • Down—Bearish

The survey is conducted weekly from Thursday, 12:00 a.m. Central Time, to Wednesday, 11:59 p.m. Central Time. AAII members participate by visiting the Sentiment Survey page (www.aaii.com/sentimentsurvey) on AAII.com and voting. The survey is open to all members, though a weekly email is sent to a rotating group of members reminding them to participate. Results of the survey are automatically tabulated by our database and published online early each Thursday morning. Prior to the year 2000, members responded by physically mailing a postcard back to the AAII offices. Response rates have varied over the years, but since the start of 2013, more than 300 AAII members take the survey on an average week.

The typical AAII member is a male in his mid-60s with a bachelor’s or graduate degree. AAII members tend to be affluent with a median portfolio size in excess of $1 million. The typical member describes himself as having a moderate level of investment knowledge and engaging primarily in fundamental analysis. This said, AAII has in excess of 170,000 members and, simply due to the sheer size of our membership, there are wide variances in wealth, investment knowledge and investing styles. We also have many female members. The AAII Sentiment Survey is unique in that it conveys the attitudes of engaged, hands-on individual investors.

Analyzing the Data

A spreadsheet with the historical data of the results is available on the Sentiment Survey page (www.aaii.com/sentimentsurvey). The average readings for bullish, neutral and bearish sentiment can be found at the bottom of the spreadsheet. As of the start of May 2014, bullish sentiment averaged 38.8%, neutral sentiment averaged 30.7% and bearish sentiment averaged 30.5%. We commonly round these readings to 39.0%, 30.5% and 30.5%, respectively, when discussing the survey results. These numbers reflect all known data and, if used to determine whether the survey has historically been a contrarian indicator, would incorporate hindsight bias.

The majority of the weekly readings have fallen in a range around these averages. Approximately 68% of all bullish sentiment readings, for instance, have been between 28.5% and 49.2%. Neutral sentiment has typically ranged between 22.0% and 39.3%. Bearish sentiment has typically ranged between 20.5% and 40.5%. Statisticians refer to these ranges as normal distributions. Figure 1 shows the normal distributions for bullish sentiment over time.


Normal distribution is defined as measurements that are within one standard deviation of the average mean value. Mathematically, 68.2% of the range of observations within a set of data fall into the normal distribution. Values outside of this range are considered to be outliers, or, in statistical terms, one standard deviation or more away from the mean. The outliers are the numbers we are most interested in. An outlier in the sentiment survey implies that bullish, neutral or bearish attitudes among individual investors were at unusually high or low levels. In other words, we want to see how the market performed after the survey results gave a “signal.”

There are a two other data points included in the Sentiment Survey spreadsheet. The first is a rolling eight-week average of the bullish sentiment. The second is the current difference between bullish and bearish sentiment: the “bull-bear spread.” Both are calculated on a weekly basis. For the purposes of this analysis, I added an additional data point: a rolling eight-week average of bearish sentiment. Market returns following outlying values for all of these data points were also calculated.

The “Prevailing” Average

To exclude the impact of hindsight bias, I had to determine what would have qualified as an outlier at a given point in the past. This is because both the average and the typical range of values have changed over time. For example, at the end of 1987, bullish sentiment had an average value of 40.6% and normal distribution of 28.9% through 52.3%. An investor looking at the data on December 31, 1987, would have no way of knowing that the normal distribution was going to narrow to 26.4% to 46.3% by the end of 1999. Eliminating hindsight bias requires conducting the analysis based solely on the data that would have been available at a given point in time.

In order to accomplish this, I created what I’ll refer to as a prevailing average. This is the average value of all historical readings since the first weekly results were recorded on July 24, 1987. It is backward looking, meaning the prevailing average does not consider any values registered after a particular point in time. It also differs from a moving average in that it has a set starting date—all data from July 1987 is always considered. In contrast, a moving average, such as the types commonly plotted on stock price charts, has a beginning date that is consistently shifted forward.

I’ll use bearish sentiment to show how the prevailing average works. On December 27, 1991, bearish sentiment had a prevailing average of 30.6%. This number is the average of all bearish sentiment readings between July 1987 and December 1991. Analysis of the survey conducted at the end of 1991 would have resulted in an average value of 30.6% being calculated. If the analysis were conducted 10 years later instead, on December 27, 2001, the average for bearish sentiment would have been calculated as 27.7%. The difference between the two values is simply the inclusion of four versus 14 years of data.

A similar methodology was used for determining what qualified as an outlier. Standard deviation, the border between what is a typical reading and what is an outlier, was calculated based on all prior data up to a specific date. The 8.7% standard deviation for neutral sentiment on June 24, 2004, for example, is based on all weekly data between July 24, 1987, (when the survey started) and June 24, 2004.

Is Sentiment Associated With Market Direction?

To determine whether a correlation exists between individual investor sentiment and market direction, I looked at how the S&P 500 performed following an unusually high or low reading in the AAII Sentiment Survey. Unusually high bullish sentiment, high bull-bear spread, or low bearish and low neutral readings were considered to be correct if the market fell over the following 26-week and 52-week periods. (In other words, the S&P 500 declined following a period with too much optimism, too little uncertainty or too little pessimism.) Low bullish numbers, low bull-bear spread, or high neutral sentiment and high bearish sentiment readings were considered to be correct if stocks rose over the following six-month periods (the S&P 500 rose after a period with little optimism, high levels of uncertainty or too much pessimism). Both the eight-week bullish and the eight-week bearish average readings were measured in the same manner as bullish and bearish sentiment readings.

Table 1 shows how the S&P 500 performed whenever an indicator in the Sentiment Survey had an outlying value.

The best contrarian signal occurred not when investors were unusually optimistic or pessimistic, but rather when they described themselves as being neutral (defined by the survey as expecting stock prices to be unchanged over the next six months). Unusually high levels of neutral sentiment have been followed by a median 26-week rise in the S&P 500 of 8.6% and a median 52-week rise in the S&P 500 of 17.7%. In contrast, the S&P 500 had a median return of 5.2% and 10.7%, respectively, over all 26-week and 52-week periods throughout the survey’s history. Equally notable, the large-cap index was up 83.3% of all 26-week periods and 88.1% of all 52-week periods following an unusually high neutral sentiment reading. (The median is the point at which there are an equal number of higher and lower values. It is less subject to being skewed upward or downward by a large single data point than average values are.)

        Bull  Bear Bull/Bear S&P
  Bullish Neutral Bearish 8-Wk 8-Wk Spread 500
+1 S.D. 26-Week Returns
Average
2.7%
7.2%
4.4%
2.5%
4.8%
3.4%
4.4%
Median
3.8%
8.6%
5.6%
3.0%
6.5%
4.3%
5.2%
Count Positive
205
50
193
212
240
178
993
Count Negative
81
10
98
94
116
65
372
Total Count
286
60
291
306
356
243
1,365
Percent Contrarian
28.3%
83.3%
66.3%
30.7%
67.4%
26.7%
–1 SD 26-Wk Returns
Average
8.4%
2.1%
4.0%
8.4%
4.0%
5.3%
4.4%
Median
7.1%
3.1%
4.5%
7.3%
5.2%
6.1%
5.2%
Count Positive
110
240
119
104
126
157
993
Count Negative
21
145
44
19
50
62
372
Total Count
131
385
163
123
176
219
1,365
Percent Contrarian
84.0%
37.7%
27.0%
84.6%
28.4%
71.7%
+1 SD 52-Wk Returns
Average
5.7%
15.9%
7.4%
5.8%
8.3%
6.9%
9.1%
Median
7.9%
17.7%
11.8%
7.9%
12.4%
8.6%
10.7%
Count Positive
209
52
205
227
256
182
1,058
Count Negative
77
7
85
79
100
58
281
Total Count
286
59
290
306
356
240
1,339
Percent Contrarian
26.9%
88.1%
70.7%
25.8%
71.9%
24.2%
–1 SD 52-Wk Returns
Average
13.8%
2.9%
8.0%
13.9%
10.8%
8.7%
9.1%
Median
17.9%
7.1%
9.5%
17.6%
11.9%
14.0%
10.7%
Count Positive
112
243
124
111
142
163
1,058
Count Negative
19
142
37
12
34
55
281
Total Count
131
385
161
123
176
218
1,339
Percent Contrarian
85.5%
36.9%
23.0%
90.2%
19.3%
74.8%

Low levels of optimism have also historically been associated with good buying opportunities during the survey’s history. The S&P 500 has risen by a median of 7.1% over the 26-week periods following an unusually low level of bullish sentiment. Gains have occurred 84.0% of the time. On a 52-week basis, the S&P 500 rose by a median of 17.9%, with gains realized following 85.5% of all low bullish sentiment readings. The occurrence of a 52-week gain in the S&P 500 was even greater when the eight-week bullish average was more than one standard deviation below average.

High levels of pessimism also tended to be followed by periods of market outperformance, but not with the same frequency or magnitude. The S&P 500 rose by a median of 5.6% during the 26-week period following an unusually high bearish sentiment reading. High levels of pessimism were followed by rising stock prices 66.3% of the time.

The data shows that it has been better to buy stocks when investors do not expect good short-term returns than when they expect prices to fall. This has at least been the case over the AAII Sentiment Survey’s 27-year period. In other words, Baron Rothschild’s advice of buying “when there is blood in the streets” may not be the best guidance. Rather, the time to buy may be when investors think there is a possibility of blood pouring (or more blood pouring) onto the streets or are simply uncertain about whether Mr. Market will be in a chipper or sullen mood.

There may be a logical reason for this. High levels of neutral sentiment suggest that while investors are not optimistic about the short-term outlook for stocks, they are not fearful of owning stocks either. This implies investors are staying engaged versus avoiding stocks. Low levels of bullish sentiment imply investors are not optimistic that prices will rise, while high levels of bearish sentiment imply investors expect stock prices to fall. The seemingly subtle difference is tied to loss aversion—an investor who is worried about falling prices will be less likely to buy stocks than one who merely doesn’t expect prices to rise over the short term. The latter investor will be more willing to risk temporary lackluster performance if he thinks valuations are attractive enough or may be looking for signs that a market bottom has been established.

While there is a link between the Sentiment Survey and stock market rebounds, it does not fully hold up in reverse. Neither high levels of optimism nor low levels of bearish sentiment have routinely been followed by declines in the S&P 500. The S&P 500 has had its worst performance whenever bullish sentiment been at unusually high levels, but it still has mostly risen, with a median 26-week gain of 3.8%. Unusually low levels of bearish sentiment have been followed by periods of even better performance, with a median 26-week gain of 4.5%.

Contrasting Eliminating Hindsight With Incorporating Hindsight

Table 2 shows the synchrony between the AAII Sentiment Survey and market direction when the current long-term averages and standard deviations are used to determine if a link exists. As previously stated, I ran this second analysis to show you a comparison of how hindsight bias affects the results. Though similar conclusions are reached regardless of whether hindsight bias is eliminated or is allowed to influence the results, there are differences.

        Bull Bear Bull/Bear S&P
  Bullish Neutral Bearish 8-Wk 8-Wk Spread 500
+1 S.D. 26-Week Returns
Average
0.9%
8.1%
4.7%
0.7%
5.7%
1.9%
4.4%
Median
2.7%
8.2%
6.3%
2.5%
8.6%
3.3%
5.2%
Count Positive
137
185
153
135
166
144
993
Count Negative
75
23
68
79
73
68
372
Total Count
212
208
221
214
239
212
1,365
Percent Contrarian
35.4%
88.9%
69.2%
36.9%
69.5%
32.1%
–1 SD 26-Wk Returns
Average
7.5%
1.8%
3.8%
7.7%
4.2%
5.3%
4.4%
Median
6.8%
4.2%
4.5%
6.8%
4.7%
5.8%
5.2%
Count Positive
192
147
161
183
153
155
993
Count Negative
40
95
58
43
55
60
372
Total Count
232
242
219
226
208
215
1,365
Percent Contrarian
82.8%
39.3%
26.5%
81.0%
26.4%
72.1%
+1 SD 52-Wk Returns
Average
1.8%
15.2%
7.7%
2.0%
9.7%
3.9%
9.1%
Median
5.9%
14.0%
13.4%
5.1%
14.3%
6.9%
10.7%
Count Positive
141
187
160
143
187
148
1,058
Count Negative
71
20
60
71
52
64
281
Total Count
212
207
220
214
239
212
1,339
Percent Contrarian
33.5%
90.3%
72.7%
33.2%
78.2%
30.2%
–1 SD 52-Wk Returns
Average
12.9%
3.4%
7.6%
12.6%
10.2%
8.4%
9.1%
Median
16.3%
7.5%
8.9%
17.3%
10.5%
14.1%
10.7%
Count Positive
117
260
106
109
100
156
1,058
Count Negative
22
141
32
16
25
53
281
Total Count
139
401
138
125
125
209
1,339
Percent Contrarian
84.2%
35.2%
23.2%
87.2%
20.0%
74.6%

Basing the analysis on what we currently consider to be readings beyond the normal distribution range resulted in fewer outliers. Unusually high levels of bearish sentiment were more correlated with rising markets and were followed by lower levels of 52-week market returns than when hindsight was excluded. Conversely, incorporating hindsight basis showed unusually low levels of bullish sentiment to have worse performance and a weaker correlation with rising markets. When 52-week returns were calculated, the results were mixed.

Though the differences are interesting, the important point to remember is that by looking at the data without hindsight bias, you get a better sense of returns that would have been achieved had you used the survey as a contrarian signal at some point in the past.

Can You Use Sentiment as a Market Timing Tool?

At first glance, the numbers suggest the AAII Sentiment Survey can be used to determine buying opportunities or times to increase your allocations to stocks. The danger in thinking that is that the link is correlated, not causal. High levels of pessimism or low levels of optimism do not cause stocks prices to rebound. Rather, they are associated with periods of market turbulence. Such periods are often characterized by reduced valuations. Therefore, the AAII Sentiment Survey may work better as a prompt to determine whether a buying, selling or rebalancing opportunity exists than as an actual market timing indicator.

Furthermore, outlying sentiment readings don’t always prove to be accurate indicators of changes in market direction. During the three-week period of July 3 through July 17, 2008, bullish sentiment came in at the unusually low levels of 23.9%, 22.2% and 25.0%. During the 26-week periods following each of these readings, the S&P 500 lost 28.4%, 27.2% and 32.3%, respectively. Granted, 2008 was a very difficult period, but the magnitude of the losses shows that the Sentiment Survey is not a flawless market timing indicator.

Sample size and the timing of data is another issue. Though there has been a historical trend between unusually high levels of neutral sentiment and good market returns, the data is mostly from the first 16 years of the survey’s existence. Only one occurrence since 2003 is included in this analysis: August 1, 2013. It is too early to know what the 26-week and 52-week returns for the market will be following the unusually high levels of neutral sentiment reached in March, April and May of this year (2014).

No market timing indicator or forecaster is correct on a consistent basis. Often, the most influential catalysts on market direction are not easy to predict in advance. Plus, even if a measure (sentiment or a different one) has historically been associated with changes in market direction, an investor could wait for a considerable period of time for a signal to appear. In the process, he could miss out on profitable periods for stocks.

This is why it is more important to focus on your long-term financial goals than trying to determine where stock prices are headed over the next month, six months or 12 months.

Charles Rotblut, CFA is a vice president at AAII and editor of the AAII Journal. Follow him on Twitter at twitter.com/CharlesRAAII.


Discussion

Jim from CO posted 3 months ago:

Sadly, Mr. Rotblut, your conclusion is that the AAII Sentiment Survey is useless. Therefore, I will stop casting my opinion and assume AAII will soon drop this opinion poll as well, so as not to falsely encourage anyone that may be using the results in their investment decision making process.


Charles Rotblut from IL posted 3 months ago:

Hi Jim,

Near the end of the article, I wrote: "The AAII Sentiment Survey may work better as a prompt to determine whether a buying, selling or rebalancing opportunity exists than as an actual market timing indicator."

It's not that I think sentiment is a useless indicator, but rather I that don't think investors should buy or sell stocks merely because sentiment is high or low. If you the readings reaching unusually high or low levels, look at other indicators (fundamentals, technicals, allocations or a combination, depending on your investing style) to assess whether you want to alter your allocation to stocks.

-Charles


Nathan Busch from MN posted 3 months ago:

Jim from CO:

Sentiment and market performance is not always black and white. I read Mr. Rotblut to say that nuances and subtleties exist that allow ferreting out the future performance of the market based upon the sentiment survey if one is patient enough to think through what the numbers mean.

Nathan A. Busch


George from WY posted 3 months ago:

In my experience over the past several years, it appears as though the survey sentiment is significantly influenced by market direction at the time the respondents are submitting their weekly surveys.
If the markets are going up, the bullishness is stronger. If markets are declining, then bearishness is stronger. If the markets are trading in a narrow range neutral sentiment seems to increase.


You need to log in as a registered AAII user before commenting.
Create an account

Log In