September 4, 2010

Poll Misinterpreted, Widely Reported in MA Senate Race

This may seem to be a parochial issue for those of you not in Massachusetts but it is an excellent example of what can happen when statistical analysis is put into the hands of those unfamiliar with the meaning of terms and methods with which analysts are intimately acquainted.

With the death  of Edward M. Kennedy, the long-time U.S. Senator of Massachusetts in August 2009, an enormous void was created in the Commonwealth’s political representation. Under state law, a special election was called and will take place on January 19. Republicans see this as an opportunity to pick up a seat that was otherwise unassailable; Democrats see retaining the seat as having great symbolic as well practical value.

The first poll in the race was released last week and it showed Democrat Martha Coakley leading Republican Scott Brown by nine points. However, the news outlet that broke the story — NPR outlet WBUR — reported that because of the poll’s 4.5% margin of error the race was a statistical dead heat. That Brown would be neck-and-neck to take the seat of the legendary Kennedy was big news and the story quickly became one of the most popular on www.wbur.org and was picked up by the local newspapers and broadcast news outlets.

Of course, a 4.5% margin of error doesn’t mean that a nine-point gap is a statistical tie.

“One of the most misunderstood things in polling is the margin of sampling error,” independent pollster Scott Rasmussen, who conducted the poll in question, later later told WBUR which ran a correction.

“In reality, in polling theory, in statistical theory, when we release a result, the most likely outcome is exactly the result that we produce. If we were to do another survey, in theory, the most likely outcome is that we’d get the exact same number, but we report results with a margin of error that we say has a 95 percent level of confidence.

“That means that 19 times out of 20, the numbers will be within 4.5 points of what we are reporting. When you work your way through all the numbers, yes, it is theoretically possible that our poll could produce a result that would show the numbers identical, but it’s a pretty long shot.”

If this story is like most, the correction will get far less attention than did the original item.

The lesson here is that you had best be sure that the people to whom you release data know how to interpret it or that you provide clear guidance on the implications of your analysis — be it your CEO, a reporter, or industry analyst. Take the time to clarify — and take nothing for granted.

POSTSCRIPT: Republican Scott Brown won this election by five percentage points. While there were many factors in his victory, this observer maintains that the mis-reporting of the poll catalyzed a momentum shift by making victory seem within Brown’s grasp, energizing his supporters, and drawing more attention to the race.

Share and Enjoy:
  • Print
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • email
  • LinkedIn
  • Ping.fm
  • StumbleUpon
  • Technorati
  • Twitter

About Eric McNulty
Editorial Director Eric J. McNulty most recently served as Managing Director of Conferences for Harvard Business Publishing. In this role he was responsible for the company's global conference and event business. His primary responsibility was editorial development and he oversaw production and marketing of both virtual and in-person programs. Eric has also written for Harvard Business Review, Harvard Management Update, Marketwatch, Strategy & Innovation, the Boston Business Journal, and Worthwhile magazine. He edited Harvard Business Publishing's Innovation Alert e-newsletter for two years and has worked with such thought leaders as Clayton Christensen, Thomas Davenport, Vijay Govindarajan, Gary Hamel, Jeanne Harris, Chan Kim, and Renee Mauborgne through Harvard Business Publishing events. Prior to joining Harvard Business Publishing, Eric was principal and founder of PM Collaborative – a marketing strategy consultancy serving clients such as Infiniti Motor Corporation, Legal Sea Foods, Cybersmith, and others. Previously he served in management and marketing roles at European Travel & Life magazine, Mark Cross, and Bloomingdale's.

Speak Your Mind

*