Introduction

After months of warnings about the risks of AI in healthcare, a quieter but more consequential story is emerging. Some health systems are using artificial intelligence to meaningfully improve patient safety—catching deterioration earlier, identifying missed diagnoses, and closing long-standing care gaps.

But these successes do not prove that AI is ready for widespread deployment.

They prove something far more uncomfortable: AI only improves safety in organizations willing to redesign workflows, governance, and accountability around it. Technology alone is not the differentiator. System maturity is.

1. The Safety Gains Are Real — and Highly Conditional

Across several health systems, AI-driven tools are showing measurable safety benefits. Models trained on local clinical data are flagging malnutrition, identifying patients at risk of deterioration, and surfacing preventive screening gaps that clinicians might otherwise miss during busy encounters.

At Phoenix Children’s Hospital, AI models built in-house have helped identify children with undiagnosed malnutrition and rapidly worsening clinical conditions—triggering earlier intervention and ICU transfers that likely prevented harm.

At CHI Health, predictive tools embedded in Epic Systems workflows are increasing cancer screening rates by proactively surfacing due tests at the point of care.

These outcomes matter. They are not theoretical.

But they are also not accidental.

2. Successful AI Safety Use Cases Share One Design Choice

In every credible example, AI operates in the background, not as a decision-maker.

Clinicians remain accountable. Alerts prompt discussion, not action. Predictions trigger review, not orders. Human judgment is preserved by design.

This is deliberate. Health systems that treat AI as decision support rather than decision authority avoid the most dangerous failure mode: automation bias.

When AI is positioned as an assistant—not a replacement—it enhances attention instead of eroding it.

3. Accuracy Alone Does Not Create Safety

One of the most revealing insights from real-world deployments is that model accuracy is not the hard part.

The harder question is: What happens after the algorithm fires?

At Phoenix Children’s, alerts are reviewed in structured clinical huddles. At CHI Health, queued recommendations still require physician validation. At Eskenazi Health, dementia prediction tools are paired with explainability—clinicians are shown why a patient was flagged, not just that they were.

Without clear downstream action pathways, even accurate models increase burden rather than reduce harm.

4. Governance, Not Innovation, Is the Real Bottleneck

The most mature systems are not racing to deploy generative AI into frontline care. They are building governance first.

At Mass General Brigham, AI governance committees evaluate tools through lenses of safety, bias, accountability, and regulatory exposure—often limiting early use to low-risk applications far from direct clinical decision-making.

This caution is not resistance. It is leadership.

As Agency for Healthcare Research and Quality leaders have noted, inaccurate or biased algorithms do not just fail quietly—they actively worsen clinician workload and decision fatigue.

5. Why Most Health Systems Will Struggle to Replicate These Results

The uncomfortable truth is that many health systems lack the prerequisites that make AI safety gains possible:

Fragmented governance structures
Limited clinical informatics capacity
Overburdened frontline teams with no bandwidth for feedback loops
Vendor-driven implementations with minimal local customization

In these environments, AI risks becoming just another alert—another interruption—another contributor to burnout.

The technology does not fail. The system does.

What Comes Next

AI can improve patient safety. The evidence is no longer hypothetical.

But the systems seeing real benefit are not those moving fastest—they are those moving deliberately. They invest in governance, preserve clinician authority, demand explainability, and redesign workflows around accountability.

For healthcare leaders, the question is no longer whether AI works.

It is whether their organization is designed to use it safely.

What this means for healthcare leaders

The issues discussed here reflect deeper structural choices facing health systems today—choices around strategy, governance, operating models, and long-term sustainability.

Organizations confronting similar challenges often benefit from stepping back, clarifying priorities, and aligning strategy with execution.

Learn how we support healthcare leaders with strategic clarity, system redesign, and performance transformation.