How to successfully manage uncertainty and deliver innovation

What do space and ML programs have in common?

Quite a lot! To name a few, high uncertainty, research-oriented nature, improvisation, and many other factors. ML professionals often feel like pioneers exploring the universe for the sake of humanity, where creativity and improvisation often play a significant role.

Maybe ML initiatives are not as complex as building and launching a Starship to the Moon, but they are difficult to plan and require frequent pivoting compared to standard SWE programs. Due to this, many AI leaders struggle to deliver AI solutions within predefined constraints such as time, effort, and benefits from the result. Additionally, because of high demand for AI/ML talent, they simply cannot cope with high turnover within their teams.

In this ‘short’ article, I am sharing a few useful tips from my 15-year career of leading data engineering, data science, and ML teams and programs. It is definitely a broad topic that could be good content for a book (maybe one day?). Still, I will give you a sneak peek on how to be more successful in leading high-performing AI/ML teams that work in R&D, applied R&D, or focus on productionization.

This article is relevant for MLR/MLEs who want to be leaders, managers who want to improve their skills, senior leadership to better mentor/coach, or AI enthusiasts interested in the area of AI leadership.

Towards success since failure is not an option!

Houston, we have a problem here!

Let’s start with the challenges. Why is it so difficult?

At first glance, it may seem like any other software program or project where you have a team of engineers, define goals, apply some form of agile (scrum, sprints, daily standups, etc.), sync with them from time to time (ehh boring stuff, do I have to?), and off you go!

Yes, you may deal with the same problems as any software manager or leader, such as high demand for talent, scarcity, distributed teams, low morale, and excessive effort for coaching and mentoring. Moreover, you may feel that everyone, especially recruiters on LinkedIn, want to poach your top talent that you’ve worked hard to curate…and you blame them for the high turnover in your team.

However, the difference and the real challenge lie in how you approach research, prototyping, and development of ML solutions, and most importantly, how you manage your team. Just to mention a few key challenges related to AI initiatives:

• Uncertainty and planning — Not having clearly defined goals, high uncertainty about deliverables and their shape (what will work and how). So often, there is limited planning, estimation, and monitoring, and vague/ not quantified success criteria that prevent measuring progress and the ultimate result.
• Nonlinear progress — ML progress is irregular, and frequent pivoting is required. Whether it is an R&D or product-oriented project, applying the most suitable approach for your problem is often not obvious and requires experimentation. And sometimes, magically one day… boom, it works! (or not…).
• Innovation vs. practicality — The temptation to work on novel ML ideas, often fundamental research, applying recent advancements from papers while being detached from goals, often the applied side, end-users perspective, deployment, maintenance, and other constraints. Balance is required.
• Stakeholders — Pressure from stakeholders to hit predefined milestones, lack of understanding of ML and its R&D nature, complexity, and high investment involved. This commonly includes use cases and initiatives where the effort, such as data collection or engineering, exceeds benefits from the developed solution.
• Lack of talent onboard — Scarcity of good AI engineers is a well-known fact. You may constantly work understaffed, missing an MLR, MLE, or good Data Engineer. Often shuffling responsibilities and wearing many hats is needed. This also involves pressure to work more effectively that may have a negative impact in the long run on the team.

If you add on top of it managers/leaders who 1) have limited ML knowledge or 2) are promoted software engineers with limited to none management experience and soft skills, then…

Yes, indeed, we have a problem!

Success trajectory — how to achieve it?

Ok, but how do we deal with all these challenges? Below, you will find a few tips on how to achieve an optimal flight path towards success:

1. A good leader…. a superhero?

Definitely not, even though sometimes it might feel like it. It’s actually a team effort that involves your crew and other groups working together. Your job is to steer the ship, equip your team with the right tools, keep tabs on things, provide technical guidance, remove obstacles, predict risks, and deal with problems when they arise.

A good leader has many traits, but extremely helpful is emotional intelligence, empathy, and good communication skills.

You should define a vision and convince your team to be part of it (literally sell it!). Don’t forget that you are a coach and mentor to them, and their development and career progress also depend on you! It’s okay if you don’t have all these traits down pat yet. The key is to start working on them.

2. Right team vibe

What does it mean?

Building the right team and culture is crucial to the success of any AI/ML program. Start by creating a startup-like atmosphere that energizes your team around a common goal. Encourage socializing, inclusion, autonomy, and a culture of innovation and continuous learning. However, the most important thing is to foster ingenuity and creativity since it will help you a lot in moments of crisis and pivoting — and such situations will happen frequently.

Sam Altman , CEO of OpenAI, once said that trust, autonomy, hard work, and a good process in place are key to achieving success.

What I would add is that motivation and empowering team members to reach their full potential are essential to achieve high-performing AI/ML teams.

Don’t forget to socialize with your team! Even a weekly lunch can bring value.

3. Learning never exhausts the mind

AI is a fast-paced discipline where every week brings new papers and breakthroughs. To stay current and competitive, AI/ML teams must continuously update their skills and knowledge. Investing in the learning and development of your team will not only improve their skills but also contribute to the overall success of your ML program.

Foster a culture of experimentation and innovation, encourage team members to explore new ideas, try new techniques, and experiment with new tools.

You can do this by reserving time on a project for experimentation or holding creative sessions. By working together and sharing knowledge, team members can learn from one another’s strengths and expertise, and they can collaborate to develop new solutions to complex problems.

4. Innovation vs. practicality — The balance!

Balance is key in AI/ML, just like in everything else. You need to find a sweet spot between being innovative and practical. It may refer to:

• Working on a right problem or use case — is it going to benefit people? Does it have a longer vision? Or Is it just super cool?
• Developing approach that solves the problem or enables the use case — opt for quick win?, state-of-the-art solution? or new approach that will revolutionize AI/ML and will bring you fame!

Both are not excludable but the most important is how you approach them. For example, if you are working on applied use cases, focus on hitting your goals even if you need to use basic models. Always aim for the simplest solution. But don’t forget to reserve some resources and time for research. Work with your team to apply the latest advancements (papers with code are often helpful) or create your own method. This way, you can boost performance of the model further, use it for future projects and maybe even publish a research paper at NeurIPS, ICML, CVPR, or other good ML conference.

5. Structure but be agile

To be successful, you need a robust process that allows you to deliver quickly and iteratively. It doesn’t have to be highly structured; instead, it should be a flexible process that you can adjust based on the project type. For example, I’m a big fan of combining Agile with Waterfall (WaterScrumFall). You plan your activities on a high level (goals, milestones, releases, success criteria etc.) but iterate in weekly sprints during execution to respond to changing situations and pivot quickly if necessary.

However, you must design such a process based on your needs, other processes in place, various constraints, and your preferences or habits. But be flexible; each team and project is unique. For example, a seasoned team working on an R&D project from the office might find Kanban helpful while ditching daily standups.

The same rule applies to tools. There are many excellent tools on the market. Try them out with your team, choose the ones that work best for your work and process, and use them to support your work.

6. Plan, Plan, Plan!

You may often hear, ‘this is R&D, no need to plan, it’s all about experimenting and discovering’. Nonsense, trust me.

It’s true that uncertainty in ML projects is high, but only planning can help you understand it, plan for various scenarios, and control it. Before starting any ML project, iteration, or activity, dedicate some time for planning. I know that especially in the early stages, it’s extremely difficult since it may be unclear what will work. But planning is an exercise that will help you control the unknown, prepare for various scenarios, and mitigate risks.

Define your goals and success criteria, opt for quick wins first, and break them down into milestones or even main tasks. Don’t go too deep here; remember, you want to be flexible as well.

Take into consideration that ML progress is usually nonlinear.

Invest time in risk management. Hold on, seriously? Who does that? You should. Create a list of potential risks and a mitigation plan that you update regularly with your team. Over time, you’ll see that managing uncertainty can become more manageable.

7. Monitor, measure and communicate

But don’t get too obsessed with it!

Part of the planning process is to define success criteria and metrics that allow you to track your progress. For monitoring purposes, you can assign them to phases and milestones. They may reflect tangible aspects, such as the performance of your model, latency, or the UX use cases covered. But they can also be intangible, such as user satisfaction, which may be more important than tangible metrics in some cases.

Make sure you set realistic KPIs and be open to changing them if the project needs to pivot.

And don’t forget about communication. It’s a key factor for every successful ML activity. Keeping your stakeholders informed about your progress, risks, and problems is crucial. Weekly or bi-weekly meetings work best (even if they’re short). Educating them about ML in plain language will help you a lot.

8. Celebrate and reflect

Lastly, celebrate with your team, appreciate even small successes, hitting milestones or even getting the model to converge. It boosts morale, enhances teamwork, and keeps the team creative. it’s important to recognize your team’s hard work and keep everyone motivated. You can celebrate in any way you like — maybe grab some coffee, buy a cake or have a team dinner. And even just sending a message on chat or email can make a difference!

What about reflection? It is also a crucial activity that you should do with your team in the form of retrospective meetings or lessons learned. It is not about blaming anybody, but rather it should allow team members to step back as a unit and evaluate what went well and what could have been done differently in order to be more effective in relation to research, development, collaboration, communication, management, and other relevant aspects.

By celebrating successes and reflecting on failures, you can also foster a culture of innovation where everyone feels encouraged to take risks and try new things.

How to avoid going off course?

Try to avoid the common mistakes if you want your team (and you) to be successful. Just listing below a few of them:

• Me Myself and I — nope, this is a collaborative effort, you cannot do much just by yourself. Praise your team, not yourself!
• I can do it better and faster — are you sure? Leadership is all about delegation and trust, and working within reasonable hours.
• This is R&D, no planning is needed — really? Each and every R&D project is planned to manage uncertainty and ensure that an output is delivered. Otherwise you could go indefinitely…
• Read a cool paper, let’s do it! — focus on delivering and later try new things. Most of papers work only in certain specific conditions.
• Monthly team meetings will do the job — how will you curate your team, share common goals, and keep them motivated? Schedule short meetings but more often, 1:1s as well.
• This is John’s fault — not true. You are responsible for the team and its performance, always ask yourself what you could do better or differently to mitigate issues.
• High team attrition is normal — it may be common, but what are you doing to prevent it? Hiring and training new team members can be expensive, taking up to a year. During this time, the team’s productivity will suffer, and valuable knowledge and expertise may be lost permanently.
• Model is not converging, let’s pump more data into it — but data collection and labeling is expensive and takes time. Try to understand first why it is not working (data, approach, model and parameters) and if more data indeed will bring value.

That’s it? No more tips?

I’ve shared with you a few main ones. By following these tips, you can overcome the challenges of leading AI initiatives and ensure the success of your projects.

Remember, AI is a rapidly evolving field, and staying innovative requires a great team, dedication, planning, balancing practicality with innovation, and continuous learning.

Be humble and curious. Wishing you smooth and successful journey! 🚀

Btw, I highly recommend Gene Kranz’s book ‘Failure is not an option’. It outlines the complexity of early space programs but also emphasizes the importance of leadership, teamwork, and perseverance in the face of challenges and setbacks. Sometimes, I have the impression that with AI development, especially AGI, we are at the same stage as trying to launch the first man to the moon!

Originally Published in Towards Data Science