Taming Big Data Is Not A Technology Issue
By Bill Franks, Nov 13, 2012
One thing that has struck me recently is that most of the focus when discussing big data is upon the technologies involved. The consensus seems to be that the biggest challenge with big data is a technological one, yet I don’t believe this to be the case. Sure, there are challenges today for organizations using big data, but, I would like to submit to you that technology is not the biggest problem. In fact, technology may be one of the easiest problems to solve when it comes time to tame big data.
The fact is that there are tools and technologies out there that can handle virtually all of the big data needs of the vast majority of organizations. As of today, you can find products and solutions that do whatever you need to do with big data. Technology itself is not the problem.
Then, what are the issues? The real problems are with resource availability, skills, process change, politics, and culture. While the technologies to solve your problems may be out there just waiting for you to implement them, it isn’t quite that easy, is it? You have to get budget, you have to do an implementation, you have to get your people up to speed on how to use the tools, you have to get buy in from various stakeholders, and you have to push against a culture averse to change.
The technology is right there, but you are unable to effectively put it to work. It FEELS like a technology issue since technology is front and center. However, it is really the cultural, people, and political issues surrounding the technology that are the problem. Let me illustrate with an example.
A few weeks back, I was at the annual Teradata Partners conference having a really good discussion with someone from one of our larger customers. When I suggested to him that his struggle with big data isn’t a technology problem, he challenged me. He pointed out that the network protocols they have in place just can’t support the level of data flow required to handle the big data they need to utilize. He felt he had a very real technological issue.
I pushed back on him and asked if there weren’t widely available products that would solve his network bandwidth problem tomorrow if he were to implement them. He said that was true but that he couldn’t just go and implement them tomorrow. I then pointed out to him that he had just proven he didn’t have a technology problem. The technology exists. He knows what it is and how to put it in place. He just can’t get it in his hands.
His real problem is with a culture that doesn’t want to invest in upgrading the network at this time. His real problem is a bureaucratic budgeting process that takes a long time to navigate. His real problem is building a business case to prove the need and value of making the upgrade. All of these problems are very real for my client and are also very valid. But, they aren’t about technology. Brush aside corporate policies, give him cash, and give him purchase authority today, and his problem will be solved very quickly.
I suspect that as many readers consider the story above, they will realize that they don’t really have a technology problem either. Technology is the easy part. It exists and it can be put in place as soon as you’re ready. However, those pesky budgets, policies, skill gaps, and cultural realities get in the way of that happening.
As your organization proceeds with the process of incorporating big data, be sure to give appropriate attention to the non-technical hurdles that you’ll face along the way. Don’t mistake them for a technology problem and don’t underestimate them. These hurdles will often be much more difficult to clear than finding a set of tools and technologies that will do what you require.
A few things to keep top of mind:
- Think through the challenges you’ll face in getting your organization to effectively use the big data tools you’ve pushed so hard to acquire.
- Plan to help your organization embrace the business model and process changes that might be suggested by results that your new tools and analytics uncover within the new data sources.
The fact that the tools and technologies to tame big data are readily available today is a terrific thing and you need not lose sleep on that front. However, there may be a lot of sleep loss as you figure out how to get your organization to actually acquire, implement, and put the tools and technologies to use. The people, cultural, and political issues surrounding big data are very big and very real.
About the author
Bill Franks is IIA’s Chief Analytics Officer, where he provides perspective on trends in the analytics and big data space and helps clients understand how IIA can support their efforts and improve analytics performance. His focus is on translating complex analytics into terms that business users can understand and working with organizations to implement their analytics effectively. His work has spanned many industries for companies ranging from Fortune 100 companies to small non-profits.
Franks is the author of the book Taming The Big Data Tidal Wave (John Wiley & Sons, Inc., April, 2012). In the book, he applies his two decades of experience working with clients on large-scale analytics initiatives to outline what it takes to succeed in today’s world of big data and analytics. Franks’ second book The Analytics Revolution (John Wiley & Sons, Inc., September, 2014) lays out how to move beyond using analytics to find important insights in data (both big and small) and into operationalizing those insights at scale to truly impact a business. He is an active speaker who has presented at dozens of events in recent years. His blog, Analytics Matters, addresses the transformation required to make analytics a core component of business decisions.
Franks earned a Bachelor’s degree in Applied Statistics from Virginia Tech and a Master’s degree in Applied Statistics from North Carolina State University. More information is available at www.bill-franks.com.