Once a week or so I hear from vendors who are creating the “data scientist in a box.” They say they can use software and hardware to get rid of those pesky human data scientists. Somewhat less frequently I hear from senior managers that they want to pursue analytics without analysts. One online travel website CEO heard me speak, and told me afterward, “I like what you say about analytics. But at my company we are going to do it without analysts—using machine learning instead.”
It is, I suppose, a worthwhile objective to rely on computers and software as much as possible to make sense of data. However, I have never seen it work to eliminate humans from the process. In fact, I have observed a perfect correlation in my career between the level of analytical capability of an organization, and the number of smart humans in its employ to help analyze and understand the data. This correlation includes the online travel site I mentioned; they did adopt machine learning approaches, but were obliged to hire a large number of machine learning experts to work with the technology.
Smart organizations build smart humans into their analytical processes from the beginning. At P&G, for example, a big factor in the success of the company’s approach to business intelligence and analytics is people—specifically its 300 “embedded analysts” who work in key functions and business units. They work closely with senior executives to explain analytical results, pose new questions of the data, and be present at meetings where analytics and data are discussed. Andy Walter, the company’s head of business intelligence, is fond of the golf analogy: if you want to get better at golf, he says, a coach is much more likely to improve your game than a new set of clubs.
Even in the high-tech big data world, humans can come in pretty handy. Palantir, the somewhat secretive organization that analyzes big data for (among others) the intelligence community, certainly has a sophisticated software platform. But humans are an important part of the equation. One early employee, Shyam Sankar, notes that “The foundation of Palantir is a belief in the potential of human-computer symbiosis.” One of Palantir’s key roles reports to Sankar: the “forward-deployed engineer.” I’m not sure how many there are of these people at Palantir, but there seem to be a lot. Like the embedded analysts at P&G (what is it with these military titles?), the forward-deployed engineers work with Palantir clients to solve problems with data and analysis. There are also plenty of software engineers at Palantir.
There are no doubt ways to reduce the ratio of analysts to computers in big data and analytics, and tools like machine learning that automate—or at least semi-automate—the process of fitting statistical models to data can improve productivity. And of course there are many analytical systems that make automated recommendations, or even feed into other systems so that humans never need be involved. However, when analytics and big data are intended to support or influence human decision-making, there is almost always a need for a human analyst to interpret and consult on the results.
If you are a computer reading this post, please don’t be offended; there will always be a place for you. But if you are a human doing the reading, you should also be somewhat comforted. I suspect there will always be a place for you too.