What Is Backtesting?
Backtesting is the method by which you identify the trades a rules-based investment strategy would have generated using historical data. The motivation for backtesting is to gauge the success of a system using historical data to see if it may be worth following going forward. While, as investors, we often see the disclaimer “past performance is no guarantee of future performance,” backtesting is an imperfect means of identifying promising investment techniques.
The problem for most individual investors is that there are few backtesting resources at their disposal for testing fundamental screening strategies. In fact, backtesting is a term more frequently used in the field of technical analysis. Technical analysis is the study of price and volume behavior in securities—stocks, mutual funds, commodities, etc.—to attempt to predict future price movements.
Whereas backtesting of technical trading strategies requires only price and volume data, which is compiled by numerous data vendors, there are only a handful of vendors offering similar historical databases of company financial data. Perhaps the most widely known screening database for academic and institutional research is Compustat from Standard & Poor’s. However, this product carries with it an intuitional price tag—thousands of dollars a year—that is beyond the means of the typical individual investor. Companies that market financial databases such as Thomson Reuters, which supplies the data for AAII’s Stock Investor program, also cater to institutional clients and price their products accordingly.
Zacks Research Wizard
For individual investors looking for an all-in-one stock screening and strategy backtesting platform, Zacks Investment Research offers its Research Wizard software. Many casual investors might be scared off by the price tag, which ranges from $1,800 to $3,000 a year, depending on which backtesting database you choose. While few of us have probably ever spent that much money on a piece of software, you must consider it in the context of its functionality as well as your overall investment portfolio. For someone with a $300,000 portfolio, the annual cost of the software is only 1% of the portfolio. By comparison, the average mutual fund has an expense ratio of around 1.3%.
Company Universe
Table 1 provides an overview of the data features and functionality of Research Wizard. The Research Wizard database consists of approximately 8,300 companies traded on the New York, American, NASDAQ, Pacific, and Toronto stock exchanges. The database also includes roughly 3,000 over-the-counter (OTC) stocks, 600 ADRs (American depositary receipts), and nearly 300 Canadian-based firms.
Screening Database Content
To filter the universe of companies, Research Wizard offers over 600 separate data fields for screening. Furthermore, there are several different screening and backtesting databases from which to choose, each with varying data periods covered. Research Wizard’s Advanced Backtesting database offers screening data going back as far as five years, 20 quarters, 60 months, or 260 weeks, depending on the data field. All told, Research Wizard’s various databases have over 118,000 individual data points.
Given that Zacks research centers on analyst earnings estimates, Research Wizard also offers some of the broadest estimates-based screening of any consumer screening service. In all, about 20% of the screening variables in the program deal with consensus earnings and sales estimates—including surprise data; long-term, annual, and quarterly projected growth rates; number of analysts; and estimate revisions.
Research Wizard does allow users to screen on company data, such as financial statement data and financial ratios and multiples, although not quite to the same degree as with estimate-related data. Table 2 lists the income statement, balance sheet, and statement of cash flows data available with Research Wizard’s fundamental databases.
Other screening variables available with Research Wizard are a respectable variety of return on investment, leverage, liquidity, and valuation ratios. Most variables in these categories have screening data going back 20 quarters or 260 weeks (five years in all). A rarity found in Research Wizard is screenable short interest data (short interest, change in short interest, and short interest as a percentage of shares outstanding).
Screening
To start you on your way with stock screening, Research Wizard provides over 90 predefined “screen definitions,” or stock screens. To create your own screens, you have over 600 data fields at your disposal. You can create filters using constant numeric values as well comparisons against other data fields and industry, sector, and S&P 500 benchmarks. You can also save your custom screens for future use. Furthermore, the program allows you to combine the criteria of one or more screens, a unique feature among stock screening services. However, note that, by default, the screening results of any combo screen are the combined set of companies passing each individual screen. The passing companies list is not those companies passing all of the filters of the combined screens.
A unique search element with Research Wizard is the Historical Search option, which allows you to screen on several data periods for the same data field. For example, if you want to screen for companies that have generated positive earnings surprises for each of the last 19 fiscal quarters, you don’t have to enter in 20 lines of screening criteria. As Figure 1 shows, you create the filter EPS Surprises (%) > 0 and then specify the period between the last fiscal quarter and 19 quarters ago. Lastly, specify that you want a positive EPS surprise for every one of the quarters in the test range.
Figure 1.
Historical Search
|
|
CLICK ON IMAGE TO
SEE FULL SIZE.
|
|
Figure 2.
Item Summary Display
|
|
CLICK ON IMAGE TO
SEE FULL SIZE.
|
|
One other unique feature of Research Wizard worth mentioning is the “Item Summary” function. When creating a screen, it is important to set appropriate operators and values in order to arrive at meaningful screening results. Using the Item Summary, you can see how the companies in the entire database universe are distributed based on any numerical field in the program. Figure 2 shows the Item Summary display for the current price-earnings ratio using 12-month earnings per share. The database has a universe of 10,323 companies, but of these 6,681 have “N/A” values—either the company has negative or missing 12-month earnings per share or it does not have a current share price. For the remaining 3,642 stocks, their price-earnings ratios are distributed in a range between 0.01 and 1,400.00, with a mean (average) value of 26.78 and a median or midpoint value of 15.95. The histogram chart provides a visual representation of the price-earnings distribution among the stock universe.
Once you run a screen, you can save the list of passing companies as a Ticker List. You can then run additional screens against this Ticker List as well as generate a variety of reports using this group of companies. Furthermore, you can also combine multiple Ticker Lists without having to re-enter ticker symbols.
Custom Expressions
Users can also create custom data fields and calculations, called “expressions,” using the Calculation Expression feature. This is one of the most robust expression builders we have run across. Users can build expressions by selecting data fields, time periods, and mathematical operators and functions. However, it is disappointing to note that the program does not allow you to save custom fields you use in custom expressions. This means that you must recreate custom fields every time you wish to use them in a screen.
A calculation expression consists of several elements. Data elements or “items” are part of a specific category. The fields in Research Wizard are numbered and when you create a calculation expression, the formula only displays the field numbers, not the actual field names. Figure 3 shows a calculation for free cash flow [net income + amortization & depreciation – changes in working capital + (increases in) property, plant, & equipment]. Seeing only the field numbers makes it difficult to decipher a given calculation expression.
With calculation expressions, you can also specify the time period of a given data item, with the most current being “recent,” or you can choose a specific time period, depending on the data item you are using. Additional options include the operator category, operator and, if need be, function category and function name.
Another useful expression function that is available in Research Wizard is rank. The ranking functions allow you to classify companies within the full stock universe or within a sector or industry. There are three ranking functions available in Research Wizard—ordinal, uniform, and histogram. Ordinal ranking will rank items by their values, based on the number of tickers in the group (universe, sector, or industry). Uniform ranking places an approximately equal number of companies into fractiles, which the user can specify. Lastly, histogram ranking creates equal-length intervals based on high and low values within each time period. Histogram ranking shows the distribution of values on the selected item. If an item has a wide range of values, with outliers or extreme values, histogram ranking will have a small number of companies in the top and bottom rankings and a large number of companies in each of the middle ranks.
Figure 3.
Building a
Calculation Expression
|
|
CLICK ON IMAGE TO
SEE FULL SIZE.
|
|
Backtesting
For backtesting purposes, there are two subscription levels. For access to six years of backtesting data, the annual fee is $1,800. Research Wizard’s Advanced Backtesting module, with backtesting data going back to 2000, costs $3,000 a year.
Just like with its screening capabilities, the bulk of Research Wizard’s backtesting rests with analyzing consensus earnings and sales estimates and ratings and the changes in these data over time. The Advanced Backtesting module offers over 50,000 data points for backtesting. However, over 33,000 of these fields deal with analyst consensus figures and broker ratings. Those hoping to backtest strategies based on historical financial data or financial ratios and multiples will probably be disappointed, as backtesting data in this area is sparse.
While the backtesting databases are accessible on-line, meaning you don’t have to store the data on your on PC, users of the Advanced Backtester do have the option of downloading the backtesting database for “local” access. This means you can perform your analysis without having to be connected to the Internet. However, you will need a high-speed connection, such as a cable modem or DSL, to download the backtesting database, as it is over 800 megabtyes in size.
Figure 4.
Backtest Screen Definition Window
|
|
CLICK ON IMAGE TO
SEE FULL SIZE.
|
|
Figure 4 is the Backtest Screen Definition window in Research Wizard. The process of backtesting is very straightforward with the program—select the strategy you wish to backtest, specify the holding period (one, two, four, 12, 24 or 52 weeks), the benchmark index, and the backtesting time period. Since there is not a true one-month holding period, we selected a four-week holding period.
Once the backtest is finished, Research Wizard generates a chart with the performance of your strategy and the benchmark you selected (Figure 5). There is also a summary report, listing the number of companies, total portfolio return, benchmark return, and excess return of the portfolio for each of the holding periods in the backtest. Summary performance statistics are given, including the total compound return (% and $), average number of stocks held, average return per holding period and maximum drawdown (largest peak-to-trough loss during the entire backtest period). The data in this table can be exported to Excel for further manipulation. You can also select individual holding periods to see the list of companies held for the period and their individual returns for that period.
Figure 5.
Results of Backtest
|
|
CLICK ON IMAGE TO
SEE FULL SIZE.
|
|
The Advanced Backtesting module also offers Automated Robustness Analysis, which provides you with the “best-case” and “worst-case” scenarios for a given backtest. If you are backtesting a strategy with a monthly holding period—buying at the beginning of the month and selling at the end of the month—the robustness analysis will show you the impact of buying at a different point in the month and holding the stocks for a different four-week period. Ideally, changing the timing of your strategy’s trades will not have a significant impact on its performance. The robustness analyzer will create a chart illustrating the performance of all the study periods, or users can specify selected studies for display. Furthermore, a statistical performance table is also generated, showing the average performance of all the differing time-period studies along with the best- and worst-performing runs and the time period with which they are associated.
Beyond being able to backtest individual stock screening strategies, Advanced Backtesting also allows you to backtest multiple strategies together (combo strategies). Combo strategies, in effect, are portfolios of screening strategies. When backtesting a combo strategy, you can view the combo performance separately or with the performance of the individual strategies in the “portfolio.”
Lastly, you can backtest combo strategies to “optimize” a screening strategy or see the impact of changing certain variables on performance. You can run a test for different values of a given data point to see how each would perform. The combo backtester also allows you to see the impact of removing a piece of criterion completely from the screen.
More sophisticated investors will inevitably ask about the “survivorship bias” of these backtesting results. Survivorship bias skews performance statistics when the data of companies that no longer exist due to mergers, bankruptcies, etc. are excluded from the testing database. Failing to include these “obsolete” firms in the historical data biases the results. According to Zacks, there is a partial survivorship bias in Research Wizard’s backtesting database. However, they are working to correct the problem, which they hope to have completed within the next two months. At this point, the backtesting database does not include delisted or acquired companies prior to December 3, 2004. It is encouraging to see that Zacks is taking steps to eliminate survivorship bias from their backtesting database.
Stock Research
Anyone who has read one of my screening articles in the AAII Journal knows that I don’t advocate blindly buying the companies passing a stock screen. Stock screening merely allows you to isolate companies with similar quantifiable characteristics. It is up to you to perform the analysis necessary to help you decide whether the stock is right for your investment portfolio.
Luckily, Zacks Research Wizard offers a wealth of company research to help you make these decisions. Company Snapshot Reports are one-page summary reports, featuring company descriptions, actual and forecasted earnings, company ratings against its industry, fundamental data, broker ratings, and price charts. There are also several expanded company reports that you can link to via the Internet, including fundamental data reports and tables, business description and management discussion reports, insider trading data, financial statements, and brokerage research reports.
Conclusion
I was very impressed with the features and functionality of Research Wizard. It is well positioned as the only all-in-one screening and backtesting platform that isn’t priced beyond the reach of most individual investors.
My biggest disappointment was that the screening and backtesting was heavily geared toward estimate and ratings data. Given that Zacks is a proprietor of this data, it makes sense that their database would rely heavily on it, as they are their own data supplier. The relative lack of financial statement data along with ratios and multiples, in my opinion, reflects the high costs of offering this data in a screening or backtesting database.
While many will be drawn to Research Wizard’s array of features, others may find them to be an obstacle. Programs such as Research Wizard require a lot of time and effort to understand all the bells and whistles they offer. Realize that the program represents a not-insignificant financial investment and be willing to invest the time necessary to make it worthwhile.
Luckily, the program itself is relatively easy to navigate and the menus are pretty straightforward. While there is an on-line help system, it tends to skim over some of the program’s more unique features. This led to a lot of trial-and-error, which some may find tedious.
Overall, I do believe the price of Research Wizard is warranted, given its features and depth of backtesting data—especially if you focus on consensus estimates. For those willing to spend the time learning the program, its rewards could very well offset the cost.
