This may seem to be a parochial issue for those of you not in Massachusetts but it is an excellent example of what can happen when statistical analysis is put into the hands of those unfamiliar with the meaning of terms and methods with which analysts are intimately acquainted.
With the death of Edward M. Kennedy, the long-time U.S. Senator of Massachusetts in August 2009, an enormous void was created in the Commonwealth’s political representation. Under state law, a special election was called and will take place on January 19. Republicans see this as an opportunity to pick up a seat that was otherwise unassailable; Democrats see retaining the seat as having great symbolic as well practical value.
The first poll in the race was released last week and it showed Democrat Martha Coakley leading Republican Scott Brown by nine points. However, the news outlet that broke the story — NPR outlet WBUR — reported that because of the poll’s 4.5% margin of error the race was a statistical dead heat. That Brown would be neck-and-neck to take the seat of the legendary Kennedy was big news and the story quickly became one of the most popular on www.wbur.org and was picked up by the local newspapers and broadcast news outlets.
Of course, a 4.5% margin of error doesn’t mean that a nine-point gap is a statistical tie.
“One of the most misunderstood things in polling is the margin of sampling error,” independent pollster Scott Rasmussen, who conducted the poll in question, later later told WBUR which ran a correction.
“In reality, in polling theory, in statistical theory, when we release a result, the most likely outcome is exactly the result that we produce. If we were to do another survey, in theory, the most likely outcome is that we’d get the exact same number, but we report results with a margin of error that we say has a 95 percent level of confidence.
“That means that 19 times out of 20, the numbers will be within 4.5 points of what we are reporting. When you work your way through all the numbers, yes, it is theoretically possible that our poll could produce a result that would show the numbers identical, but it’s a pretty long shot.”
If this story is like most, the correction will get far less attention than did the original item.
The lesson here is that you had best be sure that the people to whom you release data know how to interpret it or that you provide clear guidance on the implications of your analysis — be it your CEO, a reporter, or industry analyst. Take the time to clarify — and take nothing for granted.
POSTSCRIPT: Republican Scott Brown won this election by five percentage points. While there were many factors in his victory, this observer maintains that the mis-reporting of the poll catalyzed a momentum shift by making victory seem within Brown’s grasp, energizing his supporters, and drawing more attention to the race.
