Close

Is the AAII Sentiment Survey a Contrarian Indicator?

by Charles Rotblut, CFA

Each week, we ask AAII members a simple question: Do they feel the direction of the stock market over the next six months will be up (bullish), no change (neutral) or down (bearish)?

We refer to this question as the AAII Sentiment Survey. Since we started polling our members in 1987, our survey has provided insight into the moods of individual investors.

The survey has been become a widely followed measure. Its results are circulated by various organizations and media outlets, including Barron’s and Bloomberg. I have heard directly from, and indirectly of, many market strategists, investment newsletter writers and other financial professionals who follow the survey.

Can the results signal future market direction? In an update to my colleague Wayne Thorp’s 2004 article, “Investor Sentiment as a Contrarian Indicator” (Computerized Investing, September 2004), I look at the data and give perspective. The key to remember when reviewing the data is that a variety of indicators and criteria should be considered before making a tactical change in asset allocation.

Background of the AAII Sentiment Survey

The AAII Sentiment Survey is conducted each week from Thursday 12:01 a.m. until Wednesday at 11:59 p.m. AAII members participate by visiting the Sentiment Survey page (www.aaii.com/sentimentsurvey) on AAII.com and voting. The survey is open to all members, though a weekly email is sent to a rotating group of members reminding them to participate. Results of the survey are automatically tabulated by our database and published online early each Thursday morning. Prior to the year 2000, members responded by physically mailing a postcard back to the AAII offices. Response rates vary, though we would consider fewer than 100 votes to be low and more than 350 votes to be high. We do not track long-term response rates, though during the first four months of 2013, an average of 315 members took the survey each week.

The typical AAII member is a male in his mid-60s with a bachelor’s or graduate degree. AAII members tend to be affluent with a median portfolio size in excess of $1 million. The typical member describes himself as having a moderate level of investment knowledge and engaging primarily in fundamental analysis. This said, AAII has in excess of 160,000 members and simply due to the sheer size of our membership, there are wide variances in wealth, investment knowledge and investing styles. We also have many women members. This mix makes the AAII Sentiment Survey unique in that it conveys the attitudes of active, hands-on individual investors.

Historical Sentiment Readings

Bullish sentiment has averaged 38.8% over the life of the survey. Neutral sentiment has averaged 30.5% and bearish sentiment has averaged 30.6% over the life of the survey. (We round the bullish and bearish sentiment averages to 39.0% and 30.5% when reporting the weekly results.) These numbers equate to approximately four out of 10 AAII members expecting stock prices to rise and three out of 10 expecting prices to fall over the next six months on a typical week. This bullish slant aligns with history. The S&P 500 index was at 315 when the survey was first conducted; on May 2, 2013, the index stood at 1,583.

Gain exclusive access to all of AAII.com, PLUS our market-beating Model Stock Portfolio — currently trouncing the S&P 4-to-1!

TRY US FREE for 30 DAYS!

It is important to note that these numbers are averages. Over time, the results for all three choices have swung up and down. Within these swings, however, trends have emerged, allowing us to identify what is a typical reading, what is an unusual reading and what is an extraordinary reading. The extraordinary readings have had some correlations with market reversals, as I will soon explain.

A spreadsheet tracking the weekly readings can be downloaded from the AAII Sentiment Survey webpage. This spreadsheet lists the weekly results dating back to July 1987 and continues to be updated every Thursday on AAII.com.

What Counts as a Normal Reading?

Over time, the moods of individual investors swing from optimistic to pessimistic. These swings reflect attitudes toward the direction of the stock market, the strength or weakness of the economy, the quality of earnings and other macro factors impacting individual investors’ short-term outlook for stock prices. For example, during 2003 and 2004, AAII members stayed mostly bullish, a reflection of the rebound in stock prices and expansion of the U.S. economy. Conversely, during the bear market period of November 2007 through February 2009, AAII members largely stayed pessimistic about the market’s direction.

The survey’s results can be described as a pendulum that, at times, stays on one side or the other for prolonged periods. Both bullish and bearish sentiments have, at times, stayed above their respective historical averages for several weeks or months. At other times, the sentiment pendulum has stayed close to center, with individual investors’ moods staying largely within their historical averages.

Given these characteristics, it is useful to apply statistical analysis to the historical results. In addition to tracking the weekly readings, we maintain ongoing readings of standard deviations. Standard deviation calculates the distance from the average (or mean) a particular data point is. It is best described as an arc with elongated tails on the left and right side. The majority of the readings fall near the midpoint of the arc. Unusual readings occur in the lower left and right areas of the arc, areas known as +1 and –1 standard deviation. Extraordinary readings are those that are more than two standard deviations away from the historical average. A tiny fraction of readings are on the outer edges of the tails; these readings are three standard deviations away from the average. If the weekly results were to match up perfectly with statistical theory, we would see 68.2% of all readings within the midpoint of the arc, 27.2% qualifying as being unusual, 4.2% considered to be extraordinary and just 0.3% being extreme (more than three standard deviations away from average. Reality often does not fall into clearly defined buckets, but the statistical ranges help to convey the rarity of readings that are two or three standard deviations away from average.

Table 1 shows the historical average, lowest and highest readings for the Sentiment Survey, along with the percentages at one, two and three standard deviations away from the average for each choice. As previously stated, bullish sentiment has averaged 38.8% over the life of the survey. It has a standard deviation of 10.5%. This means readings between 28.3% and 49.3% are within the normal range over the historical results. Bullish sentiment is not considered to be extraordinarily high until it surpasses 59.8%. It is not considered to be extraordinarily low until it falls below 17.8%. The record high for optimism is 75.0%, set on January 6, 2000. The record low is 12.0%, set on November 16, 1990.

  Bullish
(%)
Neutral
(%)
Bearish
(%)
Average
38.8
30.5
30.6
Maximum
75.0
62.0
70.3
Minimum
12.0
7.7
6.0
 
+3 Standard Deviation
70.3
56.8
60.9
+2 Standard Deviation
59.8
48.1
50.8
+1 Standard Deviation
49.3
39.3
40.7
 
–1 Standard Deviation
28.3
21.8
20.5
–2 Standard Deviation
17.8
13.0
10.5
–3 Standard Deviation
7.3
4.3
0.4
Numbers are rounded.

 

Neutral sentiment has averaged 30.5% over the life of the survey. It has a standard deviation of 8.8%, with typical readings ranging between 21.8% and 39.3% (numbers are rounded). The record high for neutral sentiment is 62.0%, set on June 3, 1988. The record low is 7.7%, set on October 9, 2008.

Bearish sentiment has averaged 30.6% over the life of the survey. It has a standard deviation of 10.1%, with typical readings ranging between 20.5% and 40.7%. Bearish sentiment is not considered to be extraordinary high until it surpasses 50.8%. It is not considered to be extraordinarily low until it falls below 10.5%. The record high for bearish sentiment is 70.3%, set on March 5, 2009. The record low is 6.0%, set on August 21, 1987.

Sentiment as a Contrarian Indicator

As you can see, the moods of individual investors are subject to swings and, at times, they reach extreme levels. Most of the time, the readings from our survey provide insight into the moods of individual investors, but are well within their typical historical ranges. Extraordinary readings are more interesting, however, since investor sentiment is considered to be a contrary signal. Market watchers tend to look at high (or low) levels of optimism or pessimism as a sign that sentiment has swung too far in one direction.

In order to determine whether there is a correlation between the AAII Sentiment Survey and the direction of the market, I looked at instances when bullish sentiment or bearish sentiment was one or more standard deviations away from the average. I then calculated the performance of the S&P 500 for the following 26-week (six-month) and 52-week (12-month) periods. The data for conducting this analysis is available on the Sentiment Survey spreadsheet, which not only lists the survey’s results, but also tracks weekly price data for the S&P 500 index.

Table 2 shows the data. The returns for the S&P 500 reflect price changes over six- and 12-month periods. The numbers exclude any management fees, transaction costs or other expenses an investor may have incurred by making trading decisions based on the results. Each reading above or below one standard deviation from the historical average was treated as a separate signal. This resulted in some overlap. For instance, there were four extraordinarily high bullish readings recorded between November 27, 2003, and December 26, 2003. Each reading was treated as a separate six- and 12-month period. The purpose of the analysis was to see what happened to the market following a high or low sentiment reading, not to optimize a trading strategy based on the survey’s results.

Sentiment Level Number of
Observations
Average
S&P 500
Change
(%)
Median
S&P 500
Change
(%)
% of
Periods
Correctly
Contrarian
(%)
6-Month Performance
Bullish > +3 S.D. From Mean
2.0
7.4
7.4
0.0
Bullish > +2 S.D. From Mean
44.0
-0.7
0.3
48.0
Bullish > +1 S.D. From Mean
167.0
0.8
2.9
34.0
Bullish < –1 S.D. From Mean
212.0
6.9
6.2
80.0
Bullish < –2 S.D. From Mean
16.0
14.0
17.7
100.0
 
Bearish > +3 S.D. From Mean
3.0
25.8
23.0
100.0
Bearish > +2 S.D. From Mean
50.0
2.8
5.3
60.0
Bearish > +1 S.D. From Mean
162.0
4.7
6.0
71.0
Bearish < –1 S.D. From Mean
211.0
3.8
4.5
26.0
Bearish < –2 S.D. From Mean
9.0
-5.5
-1.7
67.0
All
1,319.0
4.0
4.7
 
12-Month Performance
Bullish > +3 S.D. From Mean
2.0
3.6
3.6
50.0
Bullish > +2 S.D. From Mean
44.0
-2.0
3.6
48.0
Bullish > +1 S.D. From Mean
167.0
2.4
6.3
31.0
Bullish < –1 S.D. From Mean
206.0
12.9
14.3
84.0
Bullish < –2 S.D. From Mean
16.0
20.7
21.7
100.0
 
Bearish > +3 S.D. From Mean
3.0
35.0
25.6
100.0
Bearish > +2 S.D. From Mean
50.0
3.1
14.3
60.0
Bearish > +1 S.D. From Mean
152.0
7.1
11.8
74.0
Bearish < –1 S.D. From Mean
211.0
7.7
9.9
24.0
Bearish < –2 S.D. From Mean
9.0
-4.3
4.8
44.0
All
1,293.0
8.4
10.2
 
Based on data from July 24, 1987, to May 2, 2013. Numbers are rounded.  

 

Any analysis of a potential market signal should include an analysis of market performance itself. This is why Table 2 also shows the average performance of the S&P 500 from July 24, 1987, through May 2, 2013. During this period there were 1,319 complete six-month periods with average and median gains of 4.0% and 4.7% for the S&P 500. The S&P 500 appreciated during 71.1% of these six-month periods and declined during 28.7% of them. There were also 1,293 12-month periods, during which the S&P 500 had average and median gains of 8.4% and 10.2%, respectively. The large-cap index rose during 77.7% of all 12-month periods and declined during 22.3% of them. (Any period with a change equal to or greater than +0.1% or –0.1% was counted as an up or down period.)

Bullish Sentiment

Neither unusual nor extraordinarily high levels of optimism are highly correlated with declining stock prices when the entire survey’s history is considered. The 44 periods with bullish sentiment readings more than two standard deviations above average were followed by a six-month fall in the S&P 500 only 48% of the time. The average six-month decline was 0.7%.

  Number of Observations Average S&P 500
6-Mo Change(%)
Median
S&P 500
6-Mo Change(%)
% of
Periods
w/Trend
Reversal
Excluding 2003–2004 Bull Market
Bullish > +2 S.D. From Mean
25.0
-5.0
-6.0
68.0
All
1,214.0
3.7
4.5
 
Excluding 2007–2009 Bear Market
Bearish > +2 S.D. From Mean
26.0
14.2
15.0
96.0
All
1,247.0
5.0
5.1
 
Numbers are rounded.

 

Bullish sentiment twice exceeded three standard deviations from its historical mean. The S&P 500 was higher six months after both readings and higher 12 months later after one of the readings. Both the average and median S&P 500 index six-month gains were 7.4%. Both the average and median S&P 500 index 12-month gains were 3.6%. Conclusions should not be drawn from these extreme readings since the sample size is so small.

The failure of high optimism as a contrarian indicator can directly be linked to 2003 and 2004. During this period of economic expansion and rising home prices, the S&P 500 rose from 880 on January 2, 2003, to 1,213 on December 30, 2004. As shown in Table 3, when the 19 periods of extraordinary optimism during this period are excluded, the S&P 500 declined by an average of 5% during the six-month period following a bullish sentiment reading of more than two standard deviations above the mean. Furthermore, the S&P 500 subsequently declined 68% of the time.

Extraordinarily low levels of optimism have worked better as a contrarian signal. Bullish sentiment has been below two standard deviations of its historical mean 16 times during the survey’s history. The average and median six-month gains for the S&P 500 following these low readings were 14.0% and 17.7%, respectively. The average and median 12-month gains were 20.7% and 21.7%, respectively. The S&P 500 large-cap index rose every time after our survey indicated an extraordinarily low level of optimism.

Bearish Sentiment

Extraordinarily high levels of pessimism have a mixed record of being correlated with higher stock prices. On a six-month basis, the S&P 500 rose 60% of the time following a bearish sentiment reading more than two standard deviations above the historical mean. The average and median gains were 2.8% and 5.3%, respectively. On a 12-month basis, the S&P 500 rose 60% of the time, with an average gain of 3.1% and a median gain of 14.3%. The average increases in prices are well below the typical increases realized throughout the entire history of the survey, though the median increases are greater than the typical gains.

Bearish sentiment exceeded three standard deviations above its historical average on three occasions. Each of these occurrences was followed by large market gains. The average and median six-month gains were 25.8% and 23.0%, respectively. The average and median 12-month gains were 35.0% and 25.6%, respectively. Boosting these numbers is the record high bearish reading of 70.3% set on March 5, 2009, literally days before the bear market ended. The S&P 500 went on to rebound by 39.5% over the next six months and 56.9% over the next 12 months.

Like bullish sentiment, bearish sentiment had a period where high readings were repeatedly recorded. During the bear market period of November 2007 through February 2009, pessimism was more than two standard deviations above its historical mean 24 times. Excluding this period increases the correlation between extraordinary pessimism and six-month market gains to 96%, as shown in Table 3. The S&P 500’s six-month gain averaged 14.2%, with a median rise of 15.0% The index’s 12-month gain was an average 19.9%, with a median rise of 22.2%.

Low levels of pessimism also have a mixed correlation with future poor market performance. Bearish sentiment has been more than two standard deviations below its historical mean on nine occasions. These readings have been followed by six-month declines 67% of the time and 12-month declines 44% of the time. The average and median six month declines are 5.5% and 1.7%, respectively. The S&P 500 has declined by an average 4.3% during the 12-month period following an extraordinarily low bearish sentiment reading, but gained 4.8% when the median change is measured instead.

Sentiment’s Role as a Market Indicator

When our Sentiment Survey was started in 1987, there was little data on the moods of individual investors. Since that time, our survey has developed a following for both its insights and as its role as a potentially contrarian indicator for market direction. As the data shows, extraordinarily low levels of optimism have consistently preceded larger-than-average six- and 12-month gains in the S&P 500.

Sentiment is not a flawless contrarian indicator, however. Though unusual, bullish and bearish sentiment readings above or below one standard deviation from their historical average have a mixed record of signaling market direction. Extraordinarily high bullish sentiment and extraordinarily low bearish sentiment (two standard deviations away from the average) have generally worked well, with the exception of two notable periods.

It will be many years before we know whether the periods of 2003–2004 and November 2007–February 2009 were mere blemishes on the survey’s record as a contrarian indicator or a sign that both optimism and pessimism can remain at high levels for an extended period of time. I tend to think the latter will be the case, given long-term market history.

The failure of sentiment to work perfectly highlights two important points. Though correlations between sentiment levels and market direction have appeared in the past, the AAII Sentiment Survey does not predict future market direction. Overly optimistic and pessimistic investor attitudes are characteristics of market tops and bottoms, but they do not cause stock prices to change direction. Rather, it is changes in expectations of future earnings and economic and valuation trends that move stock prices. The timing of such changes has proven to be difficult to predict with accuracy.

This leads to my second concluding point: Never rely on a single indicator when forecasting market direction. Rather, consider a variety of factors—including prevailing valuations, economic data, Federal Reserve policy, government policies and other prevailing macro trends—and allow for a large margin of error in your forecast. As the saying attributed to John Maynard Keynes goes, the market can stay irrational longer than you can stay solvent.

Charles Rotblut, CFA is a vice president at AAII and editor of the AAII Journal. Follow him on Twitter at twitter.com/CharlesRAAII.


Discussion

Chicago AAII Sub Chapter Member from Illinois posted about 1 year ago:

Am I the only one that is embarrassed to be associated with a group where our collective intelligence is being recognized as a "contrarian indicator" i.e. wrong?

If I was not a life-member and I all I knew about AAII was the Barron's or Bloomberg sentiment article, what would make me want to join a group of market underperformers?

Perhaps there is a better marketing tool for AAII?


Expat Member from ex-NY posted about 1 year ago:

Kudos to AAII for taking a critical look at this issue. I disagree with the Chicago Chapter Member; it's not as though AAII members were alone in missing the 2009 bottom, for example -- a lot of reputable institutional investors were urging caution. GMO, for example, thought prices were attractive but predicted that downside momentum would continue to drag markets down further than it did. I've read a lot of articles where the AAII sentiment survey gets used as a contrarian indicator, and I never interpreted it as an indictment of AAII members themselves but rather assumed the survey was being taken as a representative of broad investor sentiment, ergo sentiment extremes imply that an awful lot of optimism/pessimism -- perhaps too much -- has been priced into current market prices.

Acting like you never make mistakes is not a good marketing tool. There are plenty of newsletters that tell you they could've given you 800% returns in six months. Is that good marketing?


Rick Schoenherr from Alabama posted about 1 year ago:

This stuff is a little above my pay grade. I do sense that these numbers could be analyzed in a variety of ways, none of which is absolute. I didn't learn much here


Steve from Indiana posted about 1 year ago:

Thanks for doing this study. As mentioned, the long term upward bias of the stock market distorts the performance whether contrarian or coincident. I think the survey might be more valuable now if there were a couple of time periods, maybe 6 weeks and also 6 months. Everyone is so short term oriented now.

I think many of the services that AAII provides has done a great job of educating investors over the years. For me as a lifetime member, AAII may have been my best investment.

It's not easy trying to outperform the market. Half of the hedge funds have underperformed over the last couple of years and no one talks about them being contrary indicators.


Carl from Florida posted about 1 year ago:

This says that the easy money to be made is when the indicator goes -2 sigma bullish. 100% certainty of a quick 20% gain. Comeon crash! - I want to do this!


Marc from NH posted about 1 year ago:

I take the Chicago member's point, without necessarily agreeing or disagreeing. Perhaps a take away may be that none of us is a trained bear, trained bull or trained neutral. Rather, we are a group, like it or not, sharing a certain level of commonality if only as far as desire for quality investment information/education. We all make our decisions independently, we do not invest as a group. We are all influenced by factors determined by our personal histories and environments.

Perhaps the reason groups like Bloomberg and Baron's find the survey worth the read is that we are representative of a group known to have skin in the game. A control sample if you will as opposed to a random shot in the dark based on the momentum talking heads are so fond of pontificating about. Measuring the sentiment of holders of securities necessarily produces different results than analysis of trade volume, direction and momentum especial these days with all the programmed trading.

Being considered contrarian is neither good nor bad. It is a description, an adjective. Personally, I've been called worse.


Edward from Utah posted about 1 year ago:

The present public /private banking system in the US has increased the instability of our social milieu. This has increased the number and kind of market variables and making the future more uncertain.
As money becomes a commodity and not a medium for production and consumption cash flow is less reliable as an indicator of a healthy economy.


Bob Bubala from IN posted about 1 year ago:

I am convinced the market moves at times when the Federal Reserve chairman or members past and present speak about policy. Recent moves. mostly down, have happened after and interview. Future moves discussed drastically move the market. It could be a move that is a year away but since it is going to happen will move the momentum either up or down. Volatility is prevalent during these periods.


Robert Smith from CA posted about 1 year ago:

I have always wondered about this question. I believe a better statistical tool would be to do a cross-correlation of the sentiment survey with stock market performance. Then you could measure whether the survey correlates (or negatively correlates) better with future perfomance or past performance. If I had to guess, I would guess the best correlation is with recent past performance, since sentiment probably correlates well with how the participants' portfolio has performed in recent memory.


Chris Johenning from OH posted about 1 year ago:

Only 350 of 160,000+ members responding? Maybe you should look at ways to improve response rates and make it easier to reply? A direct email without log in requirements would increase my participation....Any other ideas


Tom Milne from Cal, Ohio posted about 1 year ago:

I agree with Robert Smith. I too would like to see the actual correlation results (ie. R squared values) between sentement and performance.
Also agree with Chris J. Need to improve response sample size


Daniel Ballisty from CA posted about 1 year ago:

I didn't see this in the article - the understanding is that when investors are overtly bullish they are at that time heavily invested, and perhaps do not have as much "dry powder" to add to the climb. Hence, it becomes a contrarian indicator. It is not an issue of intelligence, but an interpretive reading of probable reduced future money flows. The reverse would be true of a bearish indicator. It would tend to indicate a good chunk of change is out of the market and on the sidelines. Plenty of fuel to make a bull run!

The small sampling size surprised me, but it is a statistically usable size to derive sentiment with a reasonable margin of error. I've often interpreted that number as representative of all II's, not necessarily AAII members only; the source being AAII gave it creditibility.


Mark from MI posted about 1 year ago:

All of this analysis was completed. The next step is to make it useful to us.
We see the numbers for %Bullish, % Neutral, & % Bearish, but based on your analysis of the historical data, what is the most likely market direction? Otherwise each member has to try to interpret the chart.
Providing the weekly investor sentiment as well as an AAII market predictor, based on the historical analysis, would be useful.


George Muzea from NV posted 5 months ago:

For help with Contrarian behavior at market tops and bottoms, i suggest you read, The Vital Few VS. The Trivial Many.


Tim Mecke from KS posted 4 months ago:

Could you translate this information to a chart as an indicator that could be overlayed on index and stock charts? If this already exists can you tell me where to find it?


Charles Rotblut from IL posted 4 months ago:

Tim,

A complete record of the data along with the S&P 500 index level can be downloaded from the Sentiment Survey page. The spreadsheet includes a chart.

-Charles


Fred Gerbracht from CA posted 3 months ago:

Do the same 350 members post their opinions/guesses?? It would interesting to see the response pattern.
Thanks,
Fred


Charles Rotblut from IL posted 3 months ago:

Hi Fred,

We send out a reminder to take the survey to a rotating group of members each week. While there may be some who take the survey every week, there are likely other respondents who change from week to week.


-Charles


Richard from CA posted 3 months ago:

Having read this article, I see that most references to the survey are broadly attributing significance that rarely is there in the data. As usual, the financial press is generating copy based on nothing to sell advertising or to provoke excess trading. Thanks for systematic look at this "indicator".

We should also look systematically at the abuse of the term "contrarian" in the financial press. As far as I know, it was originally used to indicate investing in value based on fundamental data. Value stocks have the best long-term appreciation potential and high flyers the worst because prices often have been driven too far by excesses of sentiment. Actual performance data has been used to prove this. Now I see articles advising being contrary to any "indicator" regardless of its demonstrated significance. Caveat investor.


Bill Renaud from TX posted 20 days ago:

Several years ago (in 2008), I downloaded the AAII Investor Sentiment Survey data from the AAII website and loaded it into a spreadsheet along with the S&P500 data. I weighted the Bullish value at 100%, the Neutral value at 50%, and the Bearish value at 0%, then added the 3 values together to produce a single value that varied from 0% to 100%. When plotted on a chart with the S&P500 data in a sliding window that spanned 1 year at a time, it appeared to me that the Sentiment Survey was not a "contrarian" indicator, but simply a "lagging" indicator. It simply seemed to lag the S&P500 by a week or two. I think that this may be because investors feel bullish when the market has been going up for a while, and then finally change sentiment after a downtrend has definitely started taking place. This may also be because most AAII investors do not get to travel the world and talk to corporate CEOs like the mutual fund portfolio managers do; they don't have the real "inside scoop" on future plans of corporations. They get some of their sentiment from "family news" (layoffs, new jobs or job changes, etc.), which tends to be somewhat like watching the rear-view mirror. This is why I just watch the S&P500 and only bother to fill in the Sentiment Survey when prompted by the occasional e-mail.


You need to log in as a registered AAII user before commenting.
Create an account

Log In