WHO Guidance on Large Multi-Modal Models: Introduction

Large multi-modal models (LMMs) — also called general-purpose foundation models — are a type of generative AI that can accept more than one kind of data input (text, images, genomic data, biosensor signals, and more) and produce outputs that are not limited to the input type. Chatbots built on large language models are familiar examples; health-specific LMMs may also integrate clinical records, imaging, and research data.

For Hong Kong healthcare professionals — in public hospitals, private practice, allied health, pharmacy, and administration — LMMs are already appearing in documentation tools, patient messaging pilots, and consumer health apps. WHO issued this 2024 guidance because LMMs were adopted faster than any consumer application in history, yet societies and health systems were not fully prepared for their risks.

How LMMs differ from earlier clinical AI

Unlike narrow AI models approved for one or two specific tasks (for example, a single imaging indication), LMMs are trained on diverse datasets and can be applied to many tasks — including some they were not explicitly trained for. Their conversational interfaces can mimic human communication, leading users to treat outputs as authoritative even when the model has no understanding, moral reasoning, or guarantee of correctness.

Key differences WHO highlights:

• Versatility — one model, many uses; outputs may change with prompts or over time
• Human-like presentation — increases automation bias and uncritical acceptance
• Unpredictability — even developers may not fully explain why certain responses are generated
• Opacity of training data — making bias, legal compliance, and performance hard to assess

Building on WHO's 2021 AI ethics guidance

In 2021, WHO published comprehensive guidance on AI ethics in health, arriving at six consensus principles that remain the foundation for this LMM guidance:

1. Protect autonomy
2. Promote human well-being, safety, and the public interest
3. Ensure transparency, explainability, and intelligibility
4. Foster responsibility and accountability
5. Ensure inclusiveness and equity
6. Promote AI that is responsive and sustainable

This learning path applies those principles to LMM-specific applications, risks, and governance across the AI value chain — from development through deployment in Hong Kong and global health settings.

For Hong Kong healthcare professionals

Before using any LMM tool in practice, ask whether it was designed and evaluated for your clinical context, whether patients will know AI is involved, and whether you can verify outputs against trusted sources — especially when stakes are high.

Source: WHO — Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models (2024)