Evaluating the Performance Claims of Investment Approaches

Claims about the performance of different stock investment theories are constantly being thrown about. Sometimes it is in the promotion of an investment letter or mutual fund, sometimes in a research summary of a variety of approaches.

AAII frequently summarizes the leading investment theories, and in Computerized Investing and Stock Investor Pro we provide formulas for different approaches and sample results over various time periods. We track and present the results of different screens twice a year in the AAII Journal [see the AAII Stock Screens area].

While this research can be of value, it is also important to realize its limitations so that you can place the results in the proper perspective.

As an investor, what you really want to know is how a specific investment approach is going to perform in the context of your own portfolio in the future. Obviously, this information is not going to be available. But you can look at performance research and see if it provides any insight into how the strategy might perform in the future.

How can you tell if the research is truly insightful?

There are a number of factors that will influence the accuracy of historical research in estimating the future. You should carefully examine each one when reviewing claims of investment performance.

Is there a rational underlying theory?

There are millions of events happening simultaneously in the world at the same time the stock market is trading. If you run thousands of correlations, you are bound to find events that appear to be strongly correlated to stock performance, but are simply correlated due to chance. The theories relating to football scores and future market behavior fall into this category and are coincidence, not a true relationship. A true causal relationship will have a rationale.

Over how long a period was the historical research conducted?

There are all kinds of strategies that perform well in some markets, but few that work in many different markets. It is important that a test covers at least two economic cycles, and includes both bull and bear markets. Use eight to 10 years as a minimum for a serious evaluation of any investment strategy.

Was the test performed going forward or going back?

There are several ways to test an investment strategy. First, you can develop a theory and then test it over an eight- to 10-year period. This is the best way, because it eliminates the possibility of data mining. Data mining occurs when you backtest, using the results to formulate a theory. You can always find some theory that explains past data just by refining it enough until you find strong correlations, but these are likely to be meaningless and nonviable in the future.

An advantage of testing forward is that you get to see and deal with survivorship bias. A frequent mistake that occurs with backtesting is that the researcher takes stocks that exist at the end of the test period and traces them through the previous periods—say eight years. This approach ignores all the stocks that existed eight years earlier, but no longer exist at the end of the period. This can cause a significant bias.

Were the assumptions realistic?

This may be the most serious area of distortion. 'Simplifying assumptions' often don't simplify, they distort. For instance, since the beginning of options trading, there has been research 'demonstrating' all kinds of strategies that would provide the opportunity for great gains with minimal or no loss exposure. The examples were all based on closing prices. The problem, however, is that the strategies all involved complicated transactions, and yet in the research the bid/ask spreads (the difference between buy and sell price at any given time) were not taken into account—in practice they would eat up all the potential profit. To be at all realistic, research on options strategy performance should use the bid price for sales and the ask price for buys for each of the options over the same time period.

A Forward-Looking Test

AAII's Shadow Stock Portfolio was formed, in part, because of the problems associated with historical research. I felt a going-forward test of a theory with real portfolio problems was necessary. And based on all the positive evidence about micro-cap stocks and value investing, I thought that would be the best strategy to test.

Now that the Shadow Stock Portfolio is beyond its 10th year, we can say that many of the problems of testing the theory have been avoided. But we must still acknowledge the reality that the world can change and micro-cap stocks might perform differently in the future. We have, however, avoided many of the problems of theory testing discussed here.

While all honest research helps to shed light on successful approaches to stock portfolio management, you must be aware of the weaknesses of different research methods. In addition, you have to face the reality that, in some cases, flawed research might be used intentionally to lead us astray.