One of the most public analytics’ competitions came to a conclusion yesterday when the Bellkor’s Pragmatic Chaos team was officially awarded the Netflix Prize of $1 million for improving the company’s movie recommendation engine.
Netflix reported, “Team BellKor’s Pragmatic Chaos edged out team The Ensemble with the winning submission coming just 24 minutes before the conclusion of the nearly three-year-long contest…(the entry) represents a 10.06% improvement over Cinematch’s score on the test subset at the start of the contest.”
Watch the prize announcement on CNBC. Here’s an earlier video clip with background on the prize and the strategies of the competitors.
The papers submitted to win the prize are available on the Netflix prize site with much more detail on the algorithm. We’ll be digestng those papers in the days ahead.
The $1 million prize wasn’t the big payoff for some of the participants. According to the New York Times:
“Arnab Gupta, chief executive of Opera Solutions, a data analytics company based in New York, took a small group of his leading researchers off other work for two years. “We’ve already had a $10 million payoff internally from what we’ve learned,” Mr. Gupta said.”
“Chris Volinsky, a member of BellKor, who is a scientist at AT&T Research, said Netflix “made a brilliant move by realizing that there was a research community out there that worked on these kinds of models and was starving for data.
‘Netflix had the data, but only a handful of people working on the problem.’”
So, in essence, Netflix was able to leverage its massive, clean data set in order to scale its analytics resources to beyond world-class. Netflix CEO Reed Hastings said that it was like “getting PhD’s for $1 an hour.” Contestants, at least those like Mr. Gupta, don’t care about the hourly rate because the payoff comes through secondary learning. Seems like a win-win for everyone.
But not everyone has an enlightened employer like Mr. Gupta. In fact, without a day job it would be hard to stay fed while competing for the prize. Then again, we all seem to find extra time to pursue our passions.
What do you think of this contest-based approach to research? Have you participated? Might there be applications for this in helping governments or non-profits solve major social challenges (so long as they have the data)? Will you consider getting involved in Netflix Prize 2 or a similar endeavor? Is this the innovation model of the future?
About Eric McNulty
More from the NY Times Bits blog about the affect of diversity in team composition on overall performance:
The way teams came together, especially late in the contest, and the improved results that were achieved suggest that this kind of Internet-enabled approach, known as crowdsourcing, can be applied to complex scientific and business challenges.
That certainly seemed to be a principal lesson for the winners. The blending of different statistical and machine-learning techniques “only works well if you combine models that approach the problem differently,” said Chris Volinsky, a scientist at AT&T Research and a leader of the Bellkor team. “That’s why collaboration has been so effective, because different people approach problems differently.”
Read more: http://bits.blogs.nytimes.com/2009/09/21/netflix-awards-1-million-prize-and-starts-a-new-contest/
And a bit more from Bits worth reading:
Yet the sort of sophisticated teamwork deployed in the Netflix contest, it seems, is a tricky business. Over three years, thousands of teams from 186 countries made submissions. Yet only two could breach the 10-percent hurdle. “Having these big collaborations may be great for innovation, but it’s very, very difficult,” said Greg McAlpin, a software consultant and a leader of the Ensemble. “Out of thousands, you have only two that succeeded. The big lesson for me was that most of those collaborations don’t work.”
What has your experience been?
Read more: http://bits.blogs.nytimes.com/2009/09/21/netflix-awards-1-million-prize-and-starts-a-new-contest/