For AI health tools, you have to be able to tell real performance evidence apart from marketing collateral. It’s not just an academic exercise for governance committees and clinical reviewers. It’s essential for assessing regulatory risk and keeping patients safe. Think of performance evidence as a governance artifact, it provides a verifiable record of how an AI tool actually behaves in the wild, which is the only way to get real answers about bias and compliance.
Performance Evidence as a Governance Artifact
The FDA’s Center for Devices and Radiological Health (CDRH) has clear expectations for what counts as valid performance evidence, especially for Software as a Medical Device (SaMD) that uses AI/ML. This isn’t a one-and-done thing at launch. The FDA emphasizes ongoing Real-World Performance Monitoring. Continuous oversight is important because AI/ML models degrade over time, a problem called algorithmic drift, which happens as new real-world data starts to look different from the original training data. Without strong, constant performance monitoring, an AI tool’s initial validation can become irrelevant very quickly, introducing risks nobody saw coming.
Recent data shows why this is a big deal. One study of FDA-authorized AI-enabled medical devices found that 4.8% were later recalled, a rate much higher than the typical 2% to 3% for non-AI devices. The study pointed to gaps in post-market surveillance as a major reason for these recalls JAMA study on AI/ML medical device recalls. This just proves that initial clearance doesn’t guarantee sustained safety or efficacy. That’s why strong Real-World Performance Monitoring is so critical for compliance.
Bias and Performance: Two Sides of the Same Document
Algorithmic bias and performance are linked, and they often show up in the same documents. A weak performance record torpedoes any claim of strong governance, especially when it comes to bias. It’s simple. When an AI health tool’s performance gets inconsistent or degrades in the real world, it immediately raises concerns about whether the model is working fairly for all patient populations or under different clinical scenarios. The FDA’s focus on Good Machine Learning Practice (GMLP) principles highlights that developers need to think about bias through the whole AI lifecycle, from collecting data to post-deployment monitoring. If your training data is biased, your model’s performance will be too.
For a governance committee, this means evaluating performance evidence involves scrutinizing the underlying data, the methods for ongoing learning, and the plans for catching and fixing performance drops or new biases. How can you claim strong governance if you can’t even produce a solid performance record? Without clear, verifiable performance data, assessing bias is impossible, making any governance claims just speculation. Because these issues are so integrated, a single document like a post-market surveillance report should really cover both performance metrics and any observed disparities that might indicate bias.
The Recorded Set: HeartFlow, Tempus AI, and Paige AI
Several companies in the AI health space show different approaches to demonstrating their performance and governance. HeartFlow, for example, has been central to discussions about FDA regulatory risk and how to build clinical evidence. Their materials, along with those from Tempus AI and Paige AI, are part of a recorded set of examples for Real-World Performance Monitoring. These companies all operate under FDA CDRH oversight, using pathways like 510(k) clearance, De Novo classification, or the much more demanding Premarket Approval (PMA) process, and each path has its own evidence requirements FDA guidance on AI medical device pathways.
The vast majority of AI/ML medical device clearances, 96.2% as of June 2026, have gone through the 510(k) pathway. It’s faster, but it hinges on showing “substantial equivalence” to an existing device, which isn’t always a great fit for truly novel AI. The De Novo pathway, by contrast, is for new, low-to-moderate-risk devices with no predicate and demands independent proof of safety and effectiveness. The regulatory pathway a company chooses directly influences the depth and type of performance evidence that’s available for you to review. For companies like HeartFlow, Tempus AI, and Paige AI, the way the FDA CDRH scrutinizes their performance data sets an industry precedent. How well they comply with post-market surveillance, change control plans, and GMLP indicates their real governance maturity.
The Two Critical Questions for Reviewers
When faced with a slick PDF of performance evidence for an AI health tool, governance committees and clinical reviewers should ask two questions to test the performance records and the governance claims behind them:
- Who defined the outcome? Defining a “successful” outcome for an AI model isn’t always straightforward. Is it a clinical endpoint, a surrogate marker, or just making things run more efficiently? Understanding who defined these outcomes, the developer, an independent clinical group, or a regulator, matters. An outcome defined only by the developer might create a conflict of interest or just reflect a very narrow (and self-serving) view of clinical utility. The most credible performance evidence will be completely transparent about how outcomes were defined and who was involved, ideally a diverse group of clinical stakeholders.
- Who reviewed the result? The review process for the AI’s performance data is equally important. Did the development team check their own homework, or was the data sent out for independent review by external clinical experts, academics, or a regulatory body? Independent review is what adds objectivity and credibility to the claims. It’s the difference between marketing and science. On top of that, the review methodology itself, including the statistical analysis, the protocols for detecting bias, and whether it followed clinical guidelines, must be transparent. The FDA’s own guidance on Real-World Performance Monitoring stresses the need for solid methods to collect and analyze real-world data, ensuring the review of the results is thorough and unbiased FDA Real-World Evidence guidance.
By asking these two questions every time, you can cut past the superficial metrics and get to the substance of an AI tool’s real-world performance and its governance. This approach makes performance evidence a key artifact for ensuring responsible AI deployment in healthcare, not just another piece of paper.
Frequently Asked Questions
Why is ongoing performance monitoring crucial for AI health tools?
Ongoing performance monitoring is crucial because AI/ML models can degrade over time due to algorithmic drift, where real-world data distributions shift from their original training data. Without this continuous oversight, the initial validation of an AI tool can quickly become outdated, introducing unforeseen risks and potentially compromising patient safety.
How does the recall rate of AI-enabled medical devices compare to non-AI devices, and what does this suggest?
A recent study found that 4.8% of FDA-authorized AI-enabled medical devices were subsequently recalled, which is significantly higher than the 2% to 3% recall rate for non-AI devices. This indicates that initial clearance does not guarantee sustained safety or efficacy, emphasizing the critical need for robust Real-World Performance Monitoring as a compliance criterion.
How are questions of algorithmic bias and performance linked?
Algorithmic bias and performance are inextricably linked, often manifesting within the same documentation. A thin performance record weakens claims of robust governance, especially concerning bias, as inconsistent or deteriorating performance can indicate that the model is not performing equitably across diverse patient populations. Evaluating performance evidence therefore requires scrutinizing underlying data and methodologies for continuous learning to detect and mitigate emergent biases.
What is the significance of the regulatory pathway chosen for an AI medical device?
The regulatory pathway, such as 510(k) clearance or De Novo classification, directly influences the depth and type of performance evidence available for review. While 510(k) relies on substantial equivalence to a predicate device, De Novo requires independent evidence of safety and effectiveness for novel devices, impacting the scrutiny applied to performance data by regulatory bodies.