HealMed: Multilingual Evaluation of Large Language Models in Medicine

HealMed is an expert-reviewed multilingual medical benchmark containing 1,000 aligned examples in each of nine languages, drawn from nine datasets and spanning multiple-choice question answering, natural language inference, and open-ended QA. Each translated instance underwent two-stage review by bilingual medical experts, enabling direct comparison between machine-translated and clinically revised data. Across 14 LLMs, OpenAI’s GPT-5.4 and Google DeepMind’s Gemini-3-Flash were the most accurate and multilingual-stable systems, while Anthropic’s Claude models also maintained relatively small resource gaps. Open-source families including DeepSeek, Qwen, LLaMA, and Gemma, plus specialized systems such as MedGemma and HuatuoGPT-o1, generally degraded more in lower-resource languages; medical specialization alone did not ensure robustness. The largest multiple-choice and NLI accuracy decline occurred in Zulu, at 28.9 percentage points relative to English, compared with 15.4 percentage points in Swahili, while Thai remained closer to higher-resource languages. In open-ended QA, 59.7% of Zulu responses scored below 3 on the five-point LLM-as-judge rubric. Expert revision materially altered measured performance: adjusted MCQA and NLI shifts reached 5.8 percentage points in Zulu, showing that translation artifacts can distort model rankings and apparent language gaps. HealMed’s central contribution is therefore not only broader multilingual testing, but an auditable evaluation pipeline that separates model capability from translation quality.