While statistical significance and other technical measures of performance are commonly assessed and reported, such measures aren’t enough for an analysis and associated presentation to be deemed a success. It is also necessary to determine and communicate how the results are significant from a business and practical perspective. If the sponsors of an analysis don’t find the results to be relevant, practical, and actionable, a measure like statistical significance is irrelevant.
Statistical Significance Isn’t Enough
Assume it is proven via an A/B test that a marketing team can be 99 percent confident statistically that the lift in response from a proposed offer is at least 10 percent. That sounds terrific on the surface, and most people would think that a 10 percent lift is impressive. However, without more context, that 10 percent lift and 99 percent statistical confidence level don’t provide enough information to justify any action at all. Let’s dig into why not.
Assume that the offer tested is a bonus offer that provides an additional, free item with a purchase. As a result, it costs much more to fulfill than a regular sale due to the increased product and shipping costs. It is entirely possible that getting a 10 percent lift in response may not sufficiently cover the extra costs incurred. If the extra costs aren’t covered sufficiently, then the fact that the response rate is significantly higher from a statistical perspective doesn’t matter. The result, while statistically significant, may not be significant or actionable from a business perspective. Rather, the result may be a losing approach that is dead on arrival with a businessperson.
Note that the same principle can be applied to a range of other technical metrics that are generated by an analytical process as well. Accuracy isn’t enough. A high area under the curve isn’t enough. No combination of technical measures can be utilized without also considering the business and practical viewpoints and implications.
Assessing Business Significance
To look beyond statistical significance and other technical measures to determine business significance, it is necessary to explore and answer additional questions. In the case of our bonus offer, this includes questions such as:
- What are the full costs associated with distributing and fulfilling the bonus offer?
- How much additional revenue is projected to be generated from the offer?
- Is the bonus offer consistent with the current corporate strategy and current marketing strategy?
- Are people available to make the process changes that will be required to fulfill the offer?
- Are there any regulatory or legal considerations that must be accounted for?
- Will the additional packaging required still be sized to fit current shipping process constraints?
- What type of customer is likely to take advantage of the bonus offer?
The point is that a statistically significant increase in response is but one piece of a much more complicated puzzle. Knowing that a minimum 10 percent lift will occur helps the computations related to the above questions to be better placed in context. However, there is a lot more work to do to validate the bonus offer makes sense from a business perspective before proceeding.
Statistical Significance Doesn’t Equal Certainty Either
An additional caveat to consider is that most people feel quite comfortable if they can be 95 percent or 99 percent certain statistically that an experiment worked. Keep in mind, however, that when you’re 95 percent certain you are right, there is still a 5 percent chance you are wrong. That means that one out of every 20 times you see a similar result and act, you will be making a mistake.
You must also make sure that the level of statistical certainty matches the level of risk that can be taken comfortably and affordably by the business. For example, if a company will go bankrupt with an incorrect decision, then 95 percent certainty doesn’t seem so great. Perhaps 99.9 percent or higher would be a better bar to cross. The bar for deciding if a company should put Image A or Image B at the top of a web page for the rest of the day can be set much lower than 95 percent since the cost of a wrong decision is very low. By remembering that statistical significance does not equal certainty, it is easier to quantify and manage risk appropriately.
Assess Analysis Results From Multiple Angles
It will always feel comforting to know that something is statistically significant. However, statistical significance by itself does not enable a smart business decision. Too many analytics professionals and data scientists focus far too much on statistical significance and other technical measures. That’s what we’re formally trained in and what we’re comfortable with. While we should certainly generate those technical metrics and give them consideration, they can’t be used by themselves without further investigation.
Once an analysis is done, don’t expect your recommendations to be taken seriously without clearly explaining why the results have business and practical significance as well. A business sponsor cares about those types of significance much more than statistical significance, and for good reason! To be a valuable partner, you’ve got to broaden your focus beyond technical measures like statistical significance and provide a more complete view of your results.
Note: This blog is based on content from an upcoming, but as yet unnamed, 2022 book with Wiley on effectively presenting information. It is also adapted from my book Taming The Big Data Tidal Wave, Wiley, 2012.
Originally published by the International Institute for Analytics