Today everybody is talking about predictive analytics, and it’s undeniably desirable to be able to predict the behavior of customers, employees, voters, or other sources of uncertainty. However, while predictive analytics are always difficult, they are particularly difficult in the current economic crisis. If you’re trying to predict the future behaviors of customers based on past behaviors, for example, keep in mind that customers are no longer behaving typically. The particular products, prices, promotions, and places that persuaded customers to buy in the past may no longer be effective. The fact that almost everyone agrees this is an unprecedented economic climate means that the models created during more normal times no longer apply. Of course, if you are using predictive models to predict things that don’t involve humans or the economy, you can ignore most of what I’ve said in this paragraph.
In times of dramatic change, reactive analytics may be safer and more appropriate than predictive ones. Reactive analytics result from trying something, and seeing what happens—what the reaction is. The analytics come in because you can’t be sure that what happens isn’t random variation unless you confirm its statistical significance.
Some call this approach “test and learn,” and indeed it often involves formal (or semi-formal) testing approaches. There must be a control group, and subjects are randomly assigned to the test or control groups. The number of subjects must be sufficiently large to produce high confidence that any differences between the test and control groups aren’t random.
I wrote an article about these tests and how best to perform them in the February issue of Harvard Business Review. I didn’t think to call them “reactive analytics” when I wrote the article several months ago, and I didn’t realize how appropriate they are for bizarre times like these. But even reactive analytics have their limits. If you did tests several years ago—or anytime before this crisis—you might want to perform them again. The “learn” part of “test and learn” only applies to circumstances that are similar to those when the initial testing was performed.
If what you learned from testing in the past has anything to do with human economic behavior, it’s time to re-test.