Analytics Journal: Being Smart About Agile

By Daniel Magestro, Jan 05, 2017

In the last few years I’ve observed an increase in interest and attempts to implement an “agile” methodology in business analytics projects. This interest reflects the rapid growth of agile in IT development, where TechBeacon reports that two-thirds of surveyed IT professionals are either leaning towards agile or have fully adopted agile in their companies. Less than 20% of those companies had adopted agile methods just five years ago.

For business analytics leaders and teams, the interest in agile is well-intended. Benefits of the agile approach in software development include less time gathering requirements up front, more in-flight project involvement from business stakeholders, diverse teams, an iterative approach with increased development speed, and more effective prioritization of project tasks.

The goals above are the right goals for analytics teams seeking to increase the business relevancy, timeliness and quality of their insights: multi-functional analysis teams, engaged business experts, more effective project prioritization, iterative analysis, increased speed to insights, etc. In particular, business expert participation in the analytics process might be the most important aspiration.

However, these goals can be obscured or undermined by an overly rigid application of agile techniques to analytics work. I’ve advised many International Institute for Analytics clients on the agile methodology to analytics, and in my experience, the pitfalls typically arise from two flawed assumptions about analytics work: that all analysis projects can be treated the same, and that analytics work is just like software development.

First, analytics projects indeed come in many flavors, and some analytics activities don’t lend themselves to small agile user stories or short tasks. Force-fitting analytics projects that are quite different in nature into one style of execution not only might create barriers to insights, it might hinder the right experts from focusing on the “long game” that certain analyses require.

For example, a marketing analysis aimed at optimizing the marketing budget across several channels (TV, radio, sponsorships, etc.) might be extremely suited to highly iterative, agile work, where the marketing model can be tested and refined in close partnership with the business. In contrast, a deep, exploratory analysis of salesforce effectiveness might be hindered by trying to divvy up the analysis into small agile steps and blurring the focus across multiple needs. Business involvement also might introduce biases at key points of some analysis.

Second, it’s imperative to recognize that analytics projects and software development projects have key differences, and attempting to apply strict agile methodology to analytics work can be problematic. This can be especially harrowing when central analytics teams are housed in the IT organization, and failure to “go agile” with analytics might be viewed as a failure of the analysts rather than a problematic fit for the methodology.

Applying data analytics to business problems often is an exploratory and experimental endeavor. Some elements of agile can bring discipline (e.g., time-gated releases, improved testing) to otherwise unruly or open-ended projects. And teams certainly should avoid the IT-familiar “waterfall” approach to projects to avoid not meeting the business’s needs or expectations.

But prudence should be taken when applying agile to analytics work. Consider the noble goal to work closely with business subject matter experts. Some experts either don’t understand the intent, or they can’t (or don’t want to) make the time and commit to the level of involvement required to bring the most benefits to the project. These common challenges in agile shouldn’t hold back the potential value generated by analytics insights.

My strongest recommendation: involving teams in the design process by taking a thoughtful, flexible approach to agile analytics project management and execution. Empowering team members to agree to the goals for the team’s approach to project execution is a great first step, and my hunch is that the goals will closely reflect agile principles. Then work as an analytics team to identify which aspects of agile make sense (and which don’t), which will in turn increase team engagement as an added benefit.

Above all, it takes practice to become adept at the agile methodology. Tap agile-experienced staff for advice on best practices, then adopt elements that set analytics teams up for success. In most cases, prioritize business engagement and involvement first, then target team composition, transparent project prioritization, sharing of iterative progress on a regular schedule, and ultimately increased speed to insights. Whether Scrum masters and Post-Its are used shouldn’t get in the way of setting up analytics teams and projects for success.

About the author

Author photo

Daniel Magestro, Ph.D., is the Vice President, Research Director at the International Institute for Analytics. An accomplished analytics executive with 10+ years of experience in healthcare, banking and insurance, Dan manages IIA’s robust research agenda, leads IIA’s international network of faculty experts, and advises IIA’s global community of analytics practitioners. Dan’s primary interests include effective analytics leadership and strategy, high-performing analytics teams, and the customer journey. Prior to joining IIA, Dan managed multiple analytics teams at Cardinal Health in Columbus, Ohio, where he built an analytics center of excellence to serve the company’s Pharmaceutical Division. He also held analytics roles at JP Morgan Chase, Nationwide Insurance, and Investor Analytics. Since 2010, he is Adjunct Professor at Ohio State University’s Fisher College of Business, where he teaches courses on data analysis and advises the college on analytics initiatives. Dan came to business from science; he holds a Ph.D. in nuclear physics.