AI in Action: The Pragmatic Path to Success

In the AYDIJ article, “2025 Healthcare Analytics Ecosystem: Thriving in Chaos,” one of the 2025 predictions made was that GenAI delivers tangible value. The prediction was that the proof-of-concepts and pilots in 2024 would shift to practice in 2025—demonstrating tangible and sustained value in clinical care, business operations, patients and workforce experience, and research advancement.

Let’s just say this is happening faster than I could have imagined. Not only have we witnessed a rapid shift from pilots to practice for GenAI use cases, we have seen a new generation of pilots focused on the next generation of AI, agentic AI.

Over the next few months, we will hear from practitioners and leaders at the forefront of this transformation. These leaders will share what they are doing, their pragmatic path to success and value, and the lessons they have learned.

To kick this off, I am thrilled to introduce Holly Hallman—our latest contributor to the AYDIJ series. Holly is an exceptional talent and practitioner with deep commitment to partnerships and a relentless drive to demonstrate value—with nearly two decades of experience driving innovation at the intersection of data, analytics, and clinical operations.

As Associate Administrator (AVP) of Enterprise Data, Analytics & AI at Keck Medicine of USC, she has led the strategic planning and execution of the health system’s enterprise data platform, data governance, business analytics, health information exchange, and advanced analytics initiatives.

In this article, “AI in Action: The Pragmatic Path to Success,” Holly shares what she has learned on her AI journey as ideas became pilots and pilots became practice, and practice transformed care for patients and providers alike.

—Ryan Sousa

Artificial Intelligence (AI) is reshaping industries, transforming workflows, and delivering efficiencies that were previously unimaginable. Understanding the practical applications and limitations, deciding where to focus, and measuring and communicating success are essential for leaders looking to harness AI’s potential. Let’s take a deeper dive into each of these areas.

Practical Applications and Limitations

Healthcare organizations and their vendor partners are exploring the application of AI to nearly every business, clinical and research process. Some efforts are seeing success, while others are not, due to misapplication of technology, lack of quality data, or other factors. Healthcare organizations should pay close attention to the literature on current efforts in the industry and where their peers and vendor partners are experiencing success and use that knowledge to make wise investment decisions.

Here are examples of where I have seen success in my organization in the generative AI space.

Ambient Listening

Recent advances in AI have dramatically lowered the cost of speech transcription, as well as deriving semantic meaning from that speech and summarizing it. Administrative note-takers and clinical scribes can now focus on more advanced tasks thanks to ambient listening technology. These AI-driven solutions have dramatically reduced the time and cost typically associated with manual documentation, whether in a patient’s exam room, project working session, or a boardroom meeting.

The reduction in cost and increase in speed resulting from the application of AI to these tasks have been transformative, making tasks such as summarizing patient charts or business discussions more efficient and less reliant on human labor. One organization saw an 83% reduction in visit note completion time, which led to a decreased turnaround time for claim submission and ultimately, faster reimbursement.

There are non-monetary benefits as well, such as increased employee and patient satisfaction and reduced cognitive burden with a greater focus on being present during a patient’s visit or a strategic meeting. As a patient of a physician using this technology, I have a better overall experience when she is spending her time focused on our conversation rather than documenting notes for my chart.

One caution to an organization planning to implement an ambient listening tool for providers is to choose target outcome measures carefully. Think in terms of return on efficiency. It might not be feasible to add additional patients to a provider’s schedule after implementing the tool. However, the provider will see a better work-life balance. I expand on this thinking in the section below on measuring and communicating success.

Advanced Search Capabilities

Large Language Models (LLMs) introduced the ability to search for data embedded in documents, rather than just the metadata, such as file names. Providers can find pertinent clinical information in 50-plus pages of a patient’s health record PDF in seconds. Administrative staff can more quickly find answers to questions regarding policies and procedures.

Caution in this area should be taken when configuring these capabilities and training staff to use them, since hallucinations (responses to a prompt that are materially incorrect) are possible. Further, scanned PDFs, a common document form in healthcare, require customization during the optical character recognition (OCR) phase of the process to avoid misinterpretation of the context.

Synthetic Data

Another practical application is the generation of synthetic data for experimentation and testing. Synthetic data allows organizations to conduct research and development without exposing real-world patient information or risking privacy breaches. Researchers and product developers can generate large quantities of realistic healthcare data for analysis or training purposes without going through approval processes or risk of exposing protected health information (PHI).

Data Classification

AI also excels in data classification. Data, structured and unstructured, can be analyzed, categorized, and tagged for use across various use cases.

Activities like tagging PHI in information security or identifying specific cohorts of patients for research have become much less time-intensive from a configuration standpoint.

Operational leaders now have access to a level of insight that was previously time-consuming to obtain, required specialized skills to create, or was simply not attainable. In customer service, for example, generative AI can analyze call transcripts to detect patient sentiment or categorize the reasons for calls, streamlining processes, and enhancing feedback and training modules for call center agents.

Document Creation

Using generative AI for document creation is becoming increasingly popular. Time-consuming tasks, such as writing prior authorization and appeals letters, can be streamlined with AI-driven workflows.

Many administrative leaders and clinicians have successfully adopted LLMs to draft communications, including new emails and responses. Most people are finding that it eliminates cognitive overload, especially by anticipating standard responses to patients.

However, organizations should be cautious when releasing these types of capabilities. Often, generated email responses are lengthier, leading to buried points or introduced cognitive burden on the recipient. Arm your teams with prompting tips and tricks, so staff are not overburdening their team members with more information to read than necessary.

Whether you are pursuing the categories above or others, organizations must be mindful of the limitations of AI in highly specialized or niche domains, particularly where data is sparse or not representative. For example, if the number of positive cases of a condition (like sepsis) is too small, models risk being over-trained and failing to provide predictive value. In such cases, the AI may only recognize what clinicians have already identified, rather than offering earlier insights.

Deciding Where to Invest

Armed with an understanding of where AI tools are having an impact (and where they aren’t effective), organizations face the challenge of prioritizing where to invest. This requires a strategic approach to AI deployment, focusing on areas where the technology can deliver the most value. While one implementation may not have an immediate impact and save an organization millions in costs, small enhancements to processes will gradually increase the overall efficiency of the organization.

Like any technology strategy, AI efforts should align with the business’s strategic and operational priorities. Each year, leaders within an organization should decide collectively where to devote their time and effort to meet their goals—reviewing with executive leaders and the Board for guidance and support.

That said, aligning to a business goal is not a silver bullet. The recipe for success includes a robust AI strategy, governance, and partnerships.

AI Strategy

With rapid advancements in AI, leaders are under pressure to implement AI to assist their organization. Each organization is unique, but many are facing financial constraints. AI initiatives can be capital-intensive. Avoid the pitfalls of chasing shiny objects. If your organization is not trying to be a first-to-market company, focus on use cases that have shown a positive investment return for similar healthcare systems.

Take advantage of AI solutions embedded in existing vendor products, partner with vendors to build solutions within your in-house data ecosystem, and purchase niche products only when necessary to minimize new technical debt as much as possible. For example, it will be easiest to scale the implementation of ambient listening tools embedded in your existing electronic health record or office productivity tools.

AI Governance

Have a robust AI governance intake process to ensure the solution planned for implementation has been vetted for appropriate use. In my experience, it’s easy to find engaged leaders to represent key areas, such as compliance, legal, human resources, and clinicians, when standing up a new governance structure in this popular space.

AI Partnerships

AI cannot just be an IT project, and organizational engagement is a key factor—it’s a partnership. Clinicians, operational leaders, and staff should have the aptitude for technology advancements, the willingness to explore new capabilities, and the readiness to implement workflow changes. Stakeholder partners should also have a clear business strategy and few competing priorities. They will need to be involved early and often to provide their operational expertise and feedback through development and implementation.

Focus on internal resource development opportunities like upskilling teams on foundational concepts and processes ahead of time via existing education venues.

Pilots to Practice

Realizing value from the implementation of AI requires moving from pilot projects to broader organizational rollouts.

Effective prioritization involves a careful evaluation of business problems, available data, and the potential for AI to deliver unique value. It also means setting realistic expectations with stakeholders that not every pilot or investment will yield the desired results.

Accepting failure as a possible outcome is an important part of the AI journey. Rather than succumbing to the sunk cost fallacy—feeling compelled to show a return on every investment—organizations should set clear expectations from the outset. Stakeholders must understand that experimentation is a normal part of AI adoption. Sometimes, the best outcome is learning that a particular approach does not work, allowing resources to be redirected more effectively.

Analyze, predict, and agree to the added infrastructure and resource expenses prior to starting the pilot. Some AI solutions require less technical oversight but come with a per-user license cost. Custom AI solutions generally do not have a per-user cost but require specialized skill sets to monitor and support long term.

When pilots are successful, scaling implementation can be a challenge. From the start of the pilot phase, include a stakeholder from every impacted area, where possible. Their input, early and often, will lead to a smoother transition to operations and better adoption.

Measuring and Communicating Success

Finally, successful AI adoption should be measured, and results should be communicated throughout the lifecycle of the AI product. Early in the planning phase, target outcomes should be established to assess whether AI-driven solutions are delivering the intended value—increasing productivity, reducing costs, improving outcomes, advancing research, or enriching workforce experience. Impacted stakeholders should be held accountable for the planned outcomes just as much as the development or implementation teams.

Consider long-term advantages when creating successful adoption criteria. Not all returns are recognized immediately. In our ambient listening example, a decrease in clinician turnover rate months later could be linked back to greater satisfaction and adoption of productivity tools available.

Also, look for improvements outside of the areas directly impacted. A decrease in clinician turnover rate could eventually lead to savings in human resources’ clinician recruitment efforts.

All implementations incur long-term maintenance costs of some variety. Continuously tracking the downstream impacts of these products helps an organization understand where to shift focus or where to continue to invest.

IIA recently released an interesting e-book on measuring and communicating the true value of enterprise analytics. It is a comprehensive guide designed to help analytics leaders move beyond traditional ROI models and adopt a more nuanced, business-aligned approach to measuring the value of enterprise analytics. Replace “analytics” with “AI,” and the information is still relevant. It all starts with deciding where to focus, clarity on what you want to achieve, and how you will measure success before you begin.

Transparent communication with stakeholders is essential throughout the lifecycle of the AI product. Clearly articulating what AI has achieved—and being honest about what it has not—builds trust and supports ongoing investment in innovation.

Final Thoughts

The practical side of AI is a combination of strategic prioritization, experimental and iterative implementation, and value measurement. By understanding where AI adds value, making informed choices about where to focus, embracing an experimental mindset, and clearly communicating results, organizations can unlock the potential of AI and experience administrative relief across many departments. As seen in healthcare and beyond, the greatest gains are realized not just through technology but through thoughtful leadership and a culture that is willing to explore, learn, and adapt.