GMLP Principles 6–7: Model Design & Human–AI Teams

Principle 6: Model choice tailored to data and intended use

Model choice and design must be suited to the available data and support the intended use / intended purpose. Developers should actively mitigate known risks such as:

• Overfitting
• Performance degradation
• Security risks

Clinical benefits and risks should be well understood and used to derive clinically meaningful performance goals for testing. Considerations include impact on the overall intended population and subgroups, as well as uncertainty and variability in device inputs, outputs, and clinical use conditions.

For clinicians

A high-performing model on a benchmark dataset may still be unsuitable if inputs in your department differ (e.g. scanner settings, lab assays, documentation style) or if subgroup performance is inadequate for your patients.

Principle 7: Assess the human–AI team, not the device alone

Device performance must be assessed in the intended use environment and clinical workflow, considering interactions with healthcare providers, patients, and caregivers where applicable.

Human factors must address, for example:

• User skills and expertise
• Understanding of model outputs and limitations
• Risk of overreliance on AI suggestions
• Level of device autonomy
• User error during normal use and reasonably foreseeable misuse

Why this matters in Hong Kong practice

Regulators and IMDRF emphasise that safe AI is not only about algorithm accuracy. A tool that performs well in isolation may fail when integrated into busy ward rounds, emergency departments, or multilingual documentation workflows.

Healthcare professionals share responsibility for appropriate reliance — maintaining clinical judgment, knowing when to override the model, and reporting concerns to the manufacturer.

Source: IMDRF — Good Machine Learning Practice for Medical Device Development: Guiding Principles (January 2025)