Governance, Data Protection & Trustworthy AI

Governance

SOPs implementing GxP principles on data and algorithm governance should extend to all data, models, and algorithms used for AI/ML throughout the medicinal product lifecycle. Aspects related to governance of all components, data protection compliance, and ethical standards should be documented and regularly reviewed.

Data protection

Applicants and marketing authorisation holders must ensure all personal data — including data indirectly held within AI/ML models — are stored and processed in accordance with Union data protection legislation. Processing must comply with principles of lawfulness, fairness, transparency, purpose limitation, data minimisation, accuracy, storage limitation, integrity, confidentiality, accountability, and data protection by design and default.

A specific risk assessment focusing on the AI system should address impact on data subjects' rights and freedoms. The necessity assessment should consider anonymised or synthetic data or differential privacy; otherwise justify why these are not feasible. Proportionality should identify the least intrusive methods to minimise impact on data subjects.

Integrity aspects

High-parameter model representations can contain subject-level information similar to training data with limited insight into the representation. If personal data were used for training, evaluate whether information could be extracted through membership, inference, or model inversion attacks.

Large language models are at particular risk of memorisation due to storage capacity. Overfitting increases memorisation risk; regularisation, dropout, and random noise can provide partial to complete anonymisation. If training data are not fit for sharing, integrity-preserving measures should be taken before transferring models to less secure environments.

Trustworthy AI and conclusion

Ethical principles for trustworthy AI apply across the medicinal product lifecycle. The Assessment List for Trustworthy AI (ALTAI) from the EU High-Level Expert Group on AI can guide implementation. Key principles include:

• Human agency and oversight
• Technical robustness and safety
• Privacy and data governance
• Transparency
• Accountability
• Societal and environmental well-being
• Diversity, non-discrimination, and fairness

A human-centric approach should guide all development and deployment. Systematic impact analysis should be conducted early, with ethical and legal expertise onboarded from the start. While AI/ML shows great promise for enhancing all phases of the medicinal product lifecycle, non-transparent high-parameter models introduce risks that must be mitigated to ensure patient safety and study integrity. Active measures must be taken to avoid bias and promote AI trustworthiness, always in compliance with legal requirements and fundamental rights.

Source: EMA — Reflection Paper on the use of Artificial Intelligence (AI) in the medicinal product lifecycle (July 2023)