HealthBench Hard
A challenging variation of HealthBench that evaluates large language models' performance and safety in healthcare through 5,000 multi-turn conversations with particularly rigorous evaluation criteria validated by 262 physicians from 60 countries
Muse Spark from Meta currently leads the HealthBench Hard leaderboard with a score of 0.428 across 6 evaluated AI models.
What HealthBench Hard measures
HealthBench Hard is a text benchmark that evaluates large language models on healthcare tasks. LLM Stats tracks 6 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.2, with the leader reaching 0.4.
Compare leaders on the best AI for healthcare leaderboards.
Publication
- Paper
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Authors
- Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, and 8 others
- Published
- arXiv
- 2505.08775
Abstract
We present HealthBench, an open-source benchmark measuring the performance and safety of large language models in healthcare. HealthBench consists of 5,000 multi-turn conversations between a model and an individual user or healthcare professional. Responses are evaluated using conversation-specific rubrics created by 262 physicians. Unlike previous multiple-choice or short-answer benchmarks, HealthBench enables realistic, open-ended evaluation through 48,562 unique rubric criteria spanning several health contexts (e.g., emergencies, transforming clinical data, global health) and behavioral dimensions (e.g., accuracy, instruction following, communication). HealthBench performance over the last two years reflects steady initial progress (compare GPT-3.5 Turbo's 16% to GPT-4o's 32%) and more rapid recent improvements (o3 scores 60%). Smaller models have especially improved: GPT-4.1 nano outperforms GPT-4o and is 25 times cheaper. We additionally release two HealthBench variations: HealthBench Consensus, which includes 34 particularly important dimensions of model behavior validated via physician consensus, and HealthBench Hard, where the current top score is 32%. We hope that HealthBench grounds progress towards model development and applications that benefit human health.
Muse Spark leads with 42.8%, followed by 
GPT OSS 120B at 30.0% and GPT-5.3 Chat at 25.9%.
Progress Over Time
Interactive timeline showing model performance evolution on HealthBench Hard
HealthBench Hard Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | Meta | — | — | — | ||
| 2 | OpenAI | 117B | 131K | $0.09 / $0.45 | ||
| 3 | OpenAI | — | 128K | $1.75 / $14.00 | ||
| 4 | OpenAI | — | 400K | $5.00 / $30.00 | ||
| 5 | OpenAI | 21B | — | — | ||
| 6 | OpenAI | — | — | — |
FAQ
Common questions about HealthBench Hard.
More evaluations to explore
Related benchmarks in the same category
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
SuperGPQA is a comprehensive benchmark that evaluates large language models across 285 graduate-level academic disciplines. The benchmark contains 25,957 questions covering 13 broad disciplinary areas including Engineering, Medicine, Science, and Law, with specialized fields in light industry, agriculture, and service-oriented domains. It employs a Human-LLM collaborative filtering mechanism with over 80 expert annotators to create challenging questions that assess graduate-level knowledge and reasoning capabilities.
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.
Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.