A company to which I’m an advisor just released its second annual survey of big data use in large companies. Most of the companies (75%) participating in the survey were large financial firms, but 20% were in healthcare and life sciences, and 5% in other industries. The results of the study indicate that big data is growing up. This is mostly a positive phenomenon, but there are some warning signs too.
On the positive side, over 90% of the almost 100 executives surveyed have some initiative underway with big data, and about a third of the respondents have a production application in place. This is a big change from last year’s survey, when most participants were experimenting with big data at best. Clearly big companies are moving to make big data a mainstream enterprise business resource.
I was also gratified to see that there is an increasing focus on big data for new product development and innovation. Most of the companies (64%) said that they were using big data for this purpose, and the same percentage agreed that they were doing “research and discovery.” A related finding of the survey is that 65% were investing in the creation of analytical sandbox environments to support data discovery. And 68% felt that “new product innovations” was the greatest value to their organization from big data, though a somewhat higher percentage (80%) thought that “better fact-based decision-making” was the greatest value which is certainly not a bad thing. I have long felt that this is the highest and best use for big data, and I am happy to see that large organizations are aggressively pursuing it.
All this indicates to me that big data is maturing and developing more value to large companies. So what’s the downside? I didn’t see it so much in the survey, but rather in discussions with companies that I’ve had in my travels. The issue is that with this grown-up resource comes grown-up responsibilities. The phrase “governance for big data” is coming up more and more in large organizations. Others include “stewardship” and “control.”
“Model management,” a concept involving keeping track of analytical models that has been around for a while but was not widely adopted, is being taken seriously in financial services firms—largely because of regulatory pressures. And although it didn’t show up much in the survey, I hear a lot of people talking about the ROI that big data projects must demonstrate ahead of investment.
Now all of these are important hygiene factors for big data to aspire to. And it’s natural that as big data and analytics come out of the back room and enter the front lines of customer relationships, recommendations for employees, and regulatory oversight, we’d pay more attention to getting the details right.
Remember, though, that there is nothing more stifling to an exciting new movement than discussions about governance and control. Nothing throws cold water on a conceptual fire better than a committee for data and model ownership or stewardship. These things need to happen, but they should be imposed slowly and carefully.
Big data has revolutionary potential. It can reshape almost every aspect of how we do business. At some point it needs controls. But be wary of premature ones. You can’t impose governance until there is something well-established to govern. Big data is still in its adolescence, and we know how little interest preteens and teens have in discussing the rules of good behavior. A good parent imposes constraints slowly and subtly, and we need to take the same approach to big data.