Backtesting Is Finance’s Version of Retrospective Research

A lot of effort and money is spent looking for ways to make money in the stock market. While you can get perfectly good investment returns through index fund investing, everyone is looking for that magic trading strategy that will help them get a little bit of extra return. After all, if you can consistently earn just 1% more in returns each year, you might end up with millions more in retirement.

What better way is there to find ways to beat the market than to study the past? If history repeats itself, then maybe the stock market does as well.

Enter backtesting.

How Backtesting Works

Backtesting is when you test a trading strategy on past stock market prices.

Here’s a simple example. Let’s say that you have a theory that the stock market tends to go up in the week after Thanksgiving. Everyone is in a really great mood after the holiday. People shop online and visit malls to buy gifts for their family and friends, which puts them in a buying mood in the stock market as well.

So you might look at the last 50 Black Fridays and Cyber Mondays and see how the Dow Jones Industrial Average or the S&P 500 has performed. If the market has gone up more often than not in the week after Thanksgiving, maybe your theory is correct and you should actually buy the S&P 500 this year.

Many technical analysis strategies are evaluated with backtesting. For example, you might think that you could make money if you bought a stock whenever it crossed above its 200-day moving average and sold whenever it crossed below its 200-day moving average. To test this theory, you’d look back at past charts to see if this trading strategy would have worked.

Traders have advanced systems to formally backtest their trading strategies. Even Fidelity (Wealth-Lab Pro) and Schwab (Strategy Tester) have backtesting software available to active traders.

Backtested Trading Strategies Always Look Good On Paper

Many new hedge funds or trading newsletters have centered their sales pitches around backtested data. Because they have no track record of making money for their clients, they rely on beautiful charts or graphs showing how their trading strategies would have worked in the past.

Does Backtesting Sound Familiar, Residents and Academic Physicians?

Backtesting occurs in medicine as well. It’s called retrospective research.

A medical student or resident is asked to look at a bunch of charts and determine whether clinical or treatment factors are associated with improved patient outcomes. For example, you might look at the charts of 100 pancreatic cancer patients treated at a single hospital and see whether patients who received radiation performed better than those who did not receive radiation.

The Weaknesses of Retrospective Research (And Backtesting)

But any physician who has done even a little bit of medical research can identify the weaknesses of retrospective research.

One issue is that many hypotheses are often tested, but only the positive results are reported. Applying the standard p < 0.05 threshold for statistical significance will lead to many false-positive results if you do not formally account for testing multiple hypotheses.

This has led to a term called p-hacking. The data science blog FiveThirtyEight has explored the problem of p-hacking in academic research, with a cool interactive graphic that allows users to quickly generate “publishable” results.

There is also significant selection bias in publicized backtested results (whether it be in a hedge fund prospectus, academic financial journal, or message board). Because the financial literature, like the medical literature, prefers positive results rather than negative results because they are more likely to be high impact, people are motivated to publicize positive results, even if they end up being spurious conclusions.

Is There Such A Thing As Prospective Research In Finance?

In medicine, retrospective research is generally considered hypothesis-generating, and it typically does not change routine clinical practice until a prospective trial can be performed. For many clinical questions, a randomized controlled clinical trial is done to confirm retrospective research, but some diseases in medicine are too rare to run a clinical trial.

Whenever a finance researcher discovers a trading strategy based on historical data, is there any way to confirm whether these results are real or spurious?

Unfortunately in finance, there’s no way to run a randomized controlled clinical trial. Ideally, you would try to validate the trading strategy prospectively (i.e. start trading with real money). Unfortunately, it can take years or even decades to truly know whether a trading strategy works, because there is so much variance in stock market returns.

That’s not to say that there is nothing that can be gleaned from past data in medicine or in finance. Some clinical questions occur too infrequently to run a randomized clinical trial, and treatment decisions are guided by retrospective research alone. Similarly, there probably have been finance strategies that have been identified through backtesting and made their discoverers very wealthy (although they probably won’t share it with you or me).

Conclusion

Be wary of backtested results that you see on the Internet and elsewhere. Understand that the usual limitations of “hypothesis-generating” retrospective research applies to backtested trading strategies in finance as well.

What do you think? Have you ever tested a trading strategy on historical data? What do you think of backtesting in finance?