FDA Regulatory Strategy

27 Jul

FDA Regulation of Artificial Intelligence/ Machine Learning in Medical Devices

Introduction

The use of Artificial Intelligence/ Machine Learning (AI/ML) will revolutionize medical devices by making them more efficient and improving their performance. AI/ ML can be defined as “techniques used to design and train software algorithms to learn from and act on data received from real world experiences”. A typical application is aiding clinicians in the detection and interpretation of abnormalities shown in radiographs.

FDA and other Regulatory agencies have developed controls that apply to today’s technology but are not capable of controlling AI/ML. Current software regulations and Guidance documents control algorithms that are static. Validation of the static algorithm is part of regulatory approval. An AI/ML algorithm violates current regulatory control because it is designed to change post validation and release into the marketplace. A changed algorithm would require a new submission for each change under current regulation.

The US FDA has approved several AI/ML products via the DeNovo process. The products were approved as if their algorithm was “locked”-capable of post release change but frozen. The FDA recognizes the need to regulate AI/ML products and has released a “discussion paper” in April of 2019: “Proposed Regulatory Framework for Modifications to Artificial Intelligence/ Machine Learning based Software as a Medical Device (SaMD)”. This document is NOT a Guidance. This approach combines elements of the FDA software Pre-cert program, Total life Cycle Approach, IMDRF risk categorization principles, FDA benefit-risk framework, ISO 62304 and the 510(k) change review requirements. The emphasis for approval will change from validation of the “final” version to approval of the software development process (TPLC approach). Regulatory focus will shift from product to developer.

Summary of the process of developing AI/ML software:

A “reference standard”, a “testing” dataset and a “training” dataset are established. The reference standard is based on expert clinician interpretation of data (truthing process). These datasets are abstracted independently from the complete database of patient information.

The “training” data set is used to “teach” the algorithm. The “testing” dataset is used to test the performance of the algorithm. The data sets must be representative of all intended users and data sources. Training and testing data sets should be from different data sources.

Following training, the algorithm is evaluated. Comparison is made between the performance of the algorithm (without participation of a clinician) and the reference standard (“standalone” test). A clinical test is then conducted. Comparison is made between the algorithm aided clinician versus the reference standard and with the clinician doing unaided evaluation versus the reference standard. Evaluation of the data is called the “scoring” process. The “score’ is used to establish performance metrics for regulatory approval.

Following release, the algorithm is exposed to new training data sets (updates) as data is accumulated in the real world and it learns from them thus improving the algorithm performance. Updates are validated, prior to release without regulatory review, with a standalone test. There are several types of AI/ML software updates: performance updates, inputs updates (new data types) and intended use updates.

Summary of the FDA Proposed Approach

The proposed approach requires:

An excellent Quality System and responsive corporate culture
Software Validation as currently regulated
Algorithm changes implemented according to pre-specified objectives and plans
Algorithm changes following pre-specified procedures and rules
Extensive monitoring of Real World performance possibly including new clinical appraisals

Evaluation of the QS (design, testing, clinical assessment) and organization is part of the “excellence appraisal”.

Algorithm changes will be made per a “pre-determined change control plan” containing “software pre-specifications” (SPS). These are descriptions of the anticipated updates. Algorithm change protocols (ACP) are described. These are the procedures to be followed so that updates work correctly and risk is managed. SPS describes what are the changes and ACP describes how those changes will be controlled.

Periodic reporting post market on updates will be required. Extensive feedback and communication with users will be required (“transparency requirement”).

Summary of How to Obtain Pre-market Approval Currently

Follow the FDA software Guidance

The DeNovo process is typically used because few predicate AI/ML devices exist

An “excellent” QS and corporate culture should be in place and demonstratable

A clinical trial proving algorithm performance as explained above will be necessary.

FDA recommends, because no other Guidance exists, to follow and adapt the Guidances issued for radiology applications:

Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Device Data- Premarket Approval (PMA) and Premarket Notification Submissions”; January 2020
Computer Assisted Detection Devices Applied to Radiology Images and Radiology Device Data Premarket Notification Submissions; July 2012

Plan a comprehensive post market program. FDA will expect program to create “confidence” in the user otherwise the user will not accept it. “Transparency” of the algorithm design, explanation of how the program learns, description of the datasets used are all necessary.

Because of the complexity and newness of this technology use of the FDA PreSub program is recommended.

FDA Regulatory Strategy