One investing expert explains why theoretical performance tests are far from definitive
It’s a well-known investing mantra: past performance does not guarantee future returns. In other words, any marketing of funds based on historical performance should be taken with a grain of salt — and that goes double if the past evidence is theoretical, according to one expert.
“One particular type of performance test that seems to have become increasingly prevalent [among investment products] is the hypothetical backtest,” said Micah Hauptman, financial services counsel for the Consumer Federation of America, in a column for the Wall Street Journal. “As long as backtesting is based on assumptions that are reasonable and clear, it can provide useful information to investors … Yet there are also significant risks to investors of relying on hypothetical backtests.”
The clearest risk, Hauptman said, was that backtesters could feed past information into a powerful computer that, after analysing and cherry-picking potentially millions of inputs and outcomes, could spit out a strategy that meets certain performance metrics over longer historical time periods.
“[W]e’ll never see the iterative and often messy process from which the ‘winning’ backtest emerged,” he said. “For example, we never get to see all of the scenarios in which the strategy would have failed to deliver on its stated objective.”
Another drawback is the fact that hypothetical backtests are often constructed using index returns, which may not reflect certain costs investors would have incurred, such as commissions and taxes. “This may not be such a big deal if the strategy uses market-capitalization-weighted indexes … and trades them infrequently,” Hauptman said. “However, if the strategy uses custom indexes that are rebalanced and traded frequently, and if the strategy comes with high management fees, the actual performance is likely to be significantly lower than the backtested performance.”
The assumptions that underpin backtests may also not be realistic considering prevailing market conditions and how the fund provider conducts its trades. As an example, Hauptman considered performance tests that “incorporate the use of certain customized derivatives” and assume “certain financing arrangements with certain fixed costs.”
“[T]he specific terms of customized derivatives transactions often are contingent on who the counterparties are,” he said. “A great financing deal that one firm might be able to get might not be available to another firm.”
Hauptman noted that the US Securities and Exchange Commission has attempted to protect investors from misleading claims by compelling firms to reveal their performance-testing methodologies and assumptions.
“But in my view, it should go a step further by concluding that firms can’t merely disclose away through boilerplate language the fact that their hypothetical backtests don’t reflect actual results, particularly if those backtests and disclosures are aimed at retail investors,” he said.
