Data Quality for Safe & Effective Health AI

Fit-for-purpose data

WHO asks developers to consider whether available data are of sufficient quality to achieve the intended purpose. Healthcare data are often incomplete, biased by care pathways, or poorly labelled — issues that are serious for AI because models amplify patterns in training data.

Key considerations (Table 3 in the source) span:

• Dataset design: splitting, volume, selection bias, variable definitions, raw vs cleaned data, augmentation, and integrity;
• Labelling: consistency, grader independence, number of graders, and correlation with patient outcomes;
• Model phases: training, tuning, verification, validation, and test sets — including static vs continually learning systems;
• Governance: access controls, risk management, privacy, interoperability, and documentation transparency.

Pre-release evaluation

Before release, rigorous evaluations should check that the system will not amplify biases and errors discussed in WHO's data quality section. Careful design and prompt troubleshooting during development can surface quality issues early and prevent harm.

Stakeholders should also work toward data ecosystems that enable responsible sharing of high-quality sources — while respecting privacy and local governance.

What clinicians should verify

When assessing an AI product for your department:

1. Is the training population representative of your patients?
2. How were labels created and quality-controlled?
3. Are limitations of raw vs cleaned data documented?
4. Is there a plan if data drift or new disease phenotypes appear?

In Hong Kong, multilingual documentation, mixed public–private coding practices, and referral patterns can differ from datasets used in other regions — raising the bar for local due diligence.

Source: WHO — Regulatory considerations on artificial intelligence for health (2023)