The International Medical Device Regulators Forum (IMDRF), which brings together the major global device regulators, recently published a draft guideline on good machine learning practices for medical device software development i. Titled Guiding Principles for Industry Consultation, it is the second AI-related document that the IMDRF’s Artificial Intelligence/Machine Learning-enabled Working Group has prepared, adding to the 2022 Machine Learning-enabled Medical Devices: Key Terms and Definitions guidance documentii.
The new draft guidance provides 10 guiding principles for medical device software manufacturers to consider and apply to their software development and post-market processes. These guiding principles are not new for industry as the FDA, Health Canada and the UK’s Medicines and Healthcare products Regulatory Agency (MHRA) published a similar guidance document on good machine learning best practices in 2021 iii>.
What are the 10 guiding principles?
In summary, the 10 guiding principles in the IMDRF document include:
- Establish a well-understood intended use and multidisciplinary expertise
- Ensure good software engineering and design control practices.
- Clinical study datasets are representative of the intended patient population.
- Training datasets are independent of test datasets.
- Selected reference standards are fit-for-purpose.
- Models are tailored to the available data and the intended use.
- Performance focused on the human-AI team in the intended use environment.
- Testing demonstrates device performance during clinically relevant conditions.
- Users are provided with clear, essential information.
- Deployed models are monitored and re-training risks are managed.
These 10 principles address the management of AI/ML software from a total product life cycle approach (TPLC) – from establishing the clinical benefits and intended use, model development design controls, testing and validation of the models, end-user considerations, to deployment and maintenance of the models. Though not mandatory, medical device manufacturers should also consider the new standard for artificial intelligence management systems, ISO/IEC 42001:2023, published in 2023. It provides a framework for all developers of AI/ML algorithms to manage the algorithms from a TPLC perspective. This new standard can aid in establishing the AI/ML software processes covered in the IMDRF guidance.
Comparison with FDA Guiding Principles
In general, the IMDRF and FDA guiding principles closely overlap except for some minor variations. The table below provides a comparison between IMDRF’s and FDA’s Guiding Principles to identify similarities and differences.
| IMDRFi | FDAiii | Similarities/Differences |
| The device’s intended use/ intended purpose is well understood, and multidisciplinary expertise is leveraged throughout the total product life cycle | Multi-disciplinary expertise is leveraged throughout TPLC | IMDRF gives explicit consideration to the device’s intended use/purpose. However, both do consider clinical workflow and the need to include multi-disciplinary expertise. |
| Good software engineering, medical device design, and security practices are implemented | Good software engineering and security practices are implemented | IMDRF adds considerations for usability and explicit mention of Quality Management Systems. |
| Clinical study participants and datasets are representative of the intended patient population | Clinical study participants and data sets are representative of the intended patient population | IMDRF emphasizes that suitable datasets are fundamental for clinical evaluations and adds details for patient population and dataset drift.
|
| Training datasets are independent of test sets | Training data sets are independent of test sets | IMDRF added that the extent of validation should be appropriate to risk.
|
| Selected reference standards are fit-for-purpose | Selected reference datasets are based upon best available methods | IMDRF emphasises a ‘fit-for-purpose’ reference standard compared to FDA’s ‘accepted, best available methods…’. FDA terminology refers to ‘reference standards’ as ‘reference datasets’. |
| Model choice and design are tailored to the available data and the intended use/ intended purpose of the device | Model design is tailored to the available data and reflects the intended use of the device | Minor differences around global and local performances versus overall and subgroup patient populations. |
| Performance is assessed with a focus on the human-AI team in the intended use environment: | Focus is placed on the performance of the human-AI team: | IMDRF provides specifics on human factors considered for human-in-the-loop AI/ML software, such as user skills, user expertise, user understanding, and user error for normal and foreseeable misuse. |
| Testing demonstrates device performance during clinically relevant conditions | Testing demonstrates device performance during clinically relevant conditions | No differences. |
| Users are provided clear, essential information | Users are provided clear, essential information | IMDRF adds ‘benefits and risks’ to be considered as information to be communicated to the end user. |
| Deployed models are monitored for performance and re-training risks are managed | Deployed models are monitored for performance and re-training risks are managed | IMDRF adds a minor reference to ‘risk-based focus’ on the use of real-world monitoring for the purposes of model maintenance and improvement. |
Ref: Good machine learning practice for medical device development, IMDRF and Good Machine Learning Practice for Medical Device Development: Guiding Principles, FDA
Key considerations
With the IMDRF publishing this new draft guidance, it highlights that global regulators are working toward harmonizing their approach and expectations from medical device manufacturers in relation to AI/ML-based software. These principles are based on both general medical device software design and development of AI/ML software development best practices.i
The key is to consider a TPLC approach to ensure that rigorous AI management processes are established early in the development phases and later continue until the end of life of the software. Of utmost importance is how and with what data models are trained and tested. Ultimately, these devices will be used in clinical practice and therefore rigour in the quality and performance of the AI/ML algorithms must be assured and maintained.i
About the author:
Yervant Chijian is Director/Team Lead, Medical Devices/IVD Australia, at PharmaLex.
Yervant provides expert technical consultancy for Medical Device regulatory compliance in major markets, ensuring efficient market access.
Yervant has spent the last 20+ years in the Medical Device field, including both manufacturing and product development. His expertise lies in Regulatory Strategic Planning, particularly in the United States, Europe, Canada, Australia, and New Zealand markets, and in Product Development and Design Controls, specifically in software, AI/ML and with active devices. Coupled with his experience in Manufacturing Processes, Quality Management Systems (ISO 13485 and MDSAP), and Product Lifecycle Management, he brings a comprehensive skill set to support clients navigate through the variety of regulatory pathways and requirements.
i IMDRF/AIWG/N73 DRAFT: 2024 Good machine learning practice for medical device development: Guiding principles
