FDA Weighs Clinician-Style Tests for Generative AI Medical Devices

The Food and Drug Administration is considering whether some generative artificial intelligence medical devices should have to demonstrate competence much as physicians do before treating patients.

The agency on Tuesday (Aug. 18) released a discussion paper seeking public input on how it should assess the risks, safety and effectiveness of generative AI-enabled devices. The paper covers premarket testing, monitoring after deployment, foundation models and agentic systems that can plan and execute multistep tasks. Comments are due Oct. 19.

The proposal responds to a basic testing problem. Traditional medical software generally performs a defined function using a predictable range of inputs and outputs. Generative AI can accept open-ended instructions, produce different answers to similar prompts and change as its underlying model, safeguards or data sources evolve.

Testing every possible interaction may be impractical. Instead, the FDA is exploring whether manufacturers could demonstrate that a completed device can competently perform its intended clinical task.

Under the possible approach, manufacturers would first conduct nonclinical benchmarking. Those tests could evaluate the device’s clinical knowledge, analytical ability, safety behavior, communication and performance across different patient groups and operating conditions.

The device would then undergo clinical confirmation to show it works with actual patients, clinicians and workflows. That step wouldn’t necessarily require a prospective clinical trial in every case. The evidence would vary according to the intended use and potential harm from an incorrect output, according to the FDA’s discussion paper.

The comparison with medical training offers a useful analogy. Regulators don’t try to anticipate every situation a physician will encounter during a career. They test underlying knowledge, observe performance in clinical settings and require continuing oversight. The FDA is examining whether a version of that model could work for AI while accounting for the different technical and legal issues surrounding medical devices.

The agency would evaluate the final device as patients and clinicians use it, rather than testing the underlying foundation model by itself. That distinction could prove important because one general-purpose model may support many products with different prompts, interfaces, safeguards and clinical uses.

Oversight would continue after approval. Possible approaches include periodic retesting, clinician review of real-world outputs and monitoring for performance deterioration. A software update or change made by a third-party foundation-model provider could trigger another assessment.

That approach could eventually affect healthcare payments. Payers and healthcare finance companies may need to know which version of an AI system performed a service, whether it stayed within its validated role and whether monitoring found a decline in performance. Those records could become part of reimbursement, contracting and liability decisions.

The concept also offers banks a useful model, although the FDA proposal imposes no requirements on financial institutions. A bank could test a fraud, underwriting or collections agent against defined operational competencies, then monitor whether it remains inside those limits. That may be more practical than attempting to prove that a general-purpose model is safe for every conceivable use.

The FDA emphasized that the paper isn’t draft guidance or a proposed policy. Still, its central idea points toward a regulatory shift: AI systems may increasingly be judged by the work they can perform reliably, the boundaries they observe and how well that competence holds up over time.

The post FDA Weighs Clinician-Style Tests for Generative AI Medical Devices appeared first on PYMNTS.com.