GenAI, Ongoing Training and Development: RAN Roundtable Peer Insights

IIA’s Research and Advisory Network (RAN) clients leverage battle-tested frameworks and an exclusive network of over 150 active practitioners and unbiased experts to plan, prioritize, and execute strategic enterprise data and analytics initiatives. We regularly check the pulse of trending topics for the RAN community and facilitate critical conversations in virtual roundtable format for peer-to-peer exchange.

In a recent roundtable discussion with IIA RAN clients, data and analytics leaders from diverse industries gathered to discuss their strategies in training the workforce—from top to bottom—on generative AI. The discussion covered a wide range of topics, from AI education frameworks to measuring the effectiveness of GenAI training and skills development.

This discussion was moderated by Rehgan Bleile, IIA Expert and Co-Founder and CEO of AlignAI. Here are the key themes and takeaways:

Frameworks for GenAI Education and Implementation in Large Enterprises

In addressing the integration of GenAI within large organizations, the group outlined a structured, multi-layered educational framework. The first layer, “Concepts,” establishes the foundational understanding of GenAI, explaining basic terms and principles through resources like Coursera or LinkedIn Learning. The “Framework” layer then builds on this by setting organizational policies and adopting standards such as the NIST AI Risk Management Framework, ensuring responsible and ethical AI usage. “Roles” are defined in the third layer, distributing responsibilities across various cross-functional teams to facilitate seamless collaboration and effective project management. The final layer, “Tools,” focuses on the practical application of AI technologies, selecting the appropriate tools for specific tasks based on the comprehensive groundwork laid by the previous layers.

This structured approach advocates for a methodical integration of AI, beginning with solid theoretical knowledge before moving to application, emphasizing the importance of governance and strategic alignment with the company's broader objectives. A case in point is the evolution of one organization's AI governance into an "AI Task Force," reflecting an adaptive strategy to the dynamic AI regulatory landscape. This evolution was facilitated by aligning with existing data governance frameworks and incorporating an ethicist role to address specific industry concerns, such as bias in healthcare. This example highlights the critical nature of having a robust framework that not only addresses technological and functional aspects but also considers the ethical implications of AI deployment.

Challenges in Assessing and Integrating GenAI

Assessing and integrating GenAI presents unique challenges distinct from traditional AI, primarily due to the lack of conventional metrics such as confusion matrices, complicating direct performance comparisons. This has led to a shift toward more subjective performance measures, like user satisfaction, particularly for natural language processing (NLP) systems where accuracy assessment may require impractical manual document reviews. Additionally, the rapid adoption of GenAI technologies stirs tensions between the drive for innovation and the necessity for stringent governance, especially in regulated sectors where data mishandling could lead to severe repercussions.

Data and analytics leaders are facing a “wild west” scenario in managing GenAI, as the decentralized and sometimes unauthorized activation of AI functionalities by various internal and external parties makes governance particularly challenging. This complexity is exacerbated by the rapid evolution of vendor offerings, where AI features are added with varying degrees of transparency. To address these issues, some organizations focus governance on specific engineering use cases, coordinating closely between IT and engineering to prioritize applications that align with core business strategies. Additionally, ongoing education and updated regulatory compliance are likened to traditional IT and privacy training, underscoring the continuous effort required to manage GenAI effectively within existing frameworks. Despite these strategies, the dynamic nature of GenAI demands frequent updates to governance practices to accommodate new developments and ensure robust oversight.

Collaboration Across Disciplines

Participants underscored the critical role of the cybersecurity team in spearheading the development and enforcement of governance policies for GenAI, navigating the intricacies between various departments. Despite security’s central role, challenges arise due to the diverse and occasionally conflicting interests within organizations. Particularly, business line leaders and IT departments have often engaged independently with vendors to advance GenAI initiatives, lacking cohesive oversight or alignment with the overall business strategy. To counteract this fragmentation, data and analytic leaders stressed the importance of adopting a business-led strategy for GenAI, as advocated in C-suite and board discussions. This strategy is designed to ensure that GenAI deployments are integrated into the broader business strategy, fostering unified governance and ensuring GenAI’s strategic use aligns with core business objectives. The ongoing effort involves rallying stakeholders across the organization to establish common ownership and a strategic direction that optimizes GenAI’s impact while managing associated risks effectively.

Training and Educational Approaches for GenAI

Data and analytics leaders are deploying a variety of strategies to elevate GenAI understanding across their workforces. Techniques range from hosting "data-inspired" sessions, which use practical demonstrations to clarify GenAI applications, to introducing structured educational frameworks that categorize data and delineate between user productivity tools and application development. These educational frameworks, developed in partnership with HR, are designed to embed GenAI competencies into senior leadership evaluations, enhancing the leadership's ability to effectively implement and utilize GenAI technologies. Moreover, some companies are actively incorporating GenAI into their existing training systems, notably through "lunch and learn" sessions focused on prompt engineering and the deployment of an internal ChatGPT for handling protected information. This approach not only promotes widespread GenAI adoption but also cultivates a practical understanding of these tools among employees. To support ongoing learning and policy adherence, organizations have established monthly councils and dedicated intranet spaces, enabling employees to continually access training resources and learn about organizational policies relevant to GenAI.

Measuring the Effectiveness of GenAI Training and Skills Development

The effectiveness of GenAI training is gauged through various methods by organizations aiming to ensure comprehensive skill integration across their workforce. One method discussed includes a competency-based framework that assesses leaders on the application and integration of new digital and data skills, beyond mere participation in training. This framework helps organizations qualitatively evaluate how leaders embody and deploy learned competencies in their roles. Moreover, practical tools like Microsoft Copilot are utilized in training programs to enhance digital literacy across all employee levels, supporting the “No One Left Behind” initiative. This initiative ensures that every employee, regardless of their technical background, gains a fundamental understanding of digital and AI tools, preparing them for more advanced functionalities. Such strategic education efforts are designed to make GenAI technologies accessible and comprehensible, establishing a solid foundation for future technological engagements within the company.