MMMLU
Progress Over Time
Interactive timeline showing model performance evolution on MMMLU
MMMLU Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Anthropic | — | — | — | ||
| 2 |  Google | — | 1.0M | $2.00 / $12.00 | ||
| 3 | Google | — | 1.0M | $0.50 / $3.00 | ||
| 3 | Google | — | — | — | ||
| 5 | Anthropic | — | 1.0M | $5.00 / $25.00 | ||
| 6 | Anthropic | — | 1.0M | $5.00 / $25.00 | ||
| 7 | Anthropic | — | — | — | ||
| 8 |  Alibaba Cloud / Qwen Team | — | 1.0M | $1.25 / $3.75 | ||
| 9 |  OpenAI | — | 400K | $1.75 / $14.00 | ||
| 10 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 10 | Anthropic | — | — | — | ||
| 12 | Anthropic | — | 1.0M | $3.00 / $15.00 | ||
| 13 | Anthropic | — | 200K | $3.00 / $15.00 | ||
| 14 | Alibaba Cloud / Qwen Team | — | — | — | ||
| 15 | Google | — | 1.0M | $0.25 / $1.50 | ||
| 16 | Anthropic | — | — | — | ||
| 17 | Alibaba Cloud / Qwen Team | 397B | 262K | $0.45 / $3.00 | ||
| 18 | Google | 31B | 262K | $0.09 / $0.34 | ||
| 19 | OpenAI | — | — | — | ||
| 20 | OpenAI | — | 1.0M | $2.00 / $8.00 | ||
| 21 | Alibaba Cloud / Qwen Team | 122B | 262K | $0.29 / $2.40 | ||
| 21 | Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 23 | Anthropic | — | — | — | ||
| 24 | Google | 25B | 262K | $0.07 / $0.34 | ||
| 25 | Anthropic | — | — | — | ||
| 26 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.26 / $2.60 | ||
| 27 |  LG AI Research | 236B | — | — | ||
| 28 |  Mistral AI | 675B | 262K | $0.50 / $1.50 | ||
| 28 | 675B | — | — | |||
| 28 | Mistral AI | 675B | — | — | ||
| 28 | 675B | — | — | |||
| 32 | Alibaba Cloud / Qwen Team | 35B | 262K | $0.14 / $1.00 | ||
| 33 | OpenAI | — | — | — | ||
| 34 | Alibaba Cloud / Qwen Team | 1.0T | 256K | $1.20 / $6.00 | ||
| 35 | OpenAI | 117B | — | — | ||
| 36 | Google | 12B | — | — | ||
| 37 | Anthropic | — | 200K | $1.00 / $5.00 | ||
| 38 |  ByteDance | — | 256K | $0.10 / $0.40 | ||
| 39 | Google | 25B | — | — | ||
| 40 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 41 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.10 / $0.15 | ||
| 42 | OpenAI | — | 1.0M | $0.40 / $1.60 | ||
| 43 | Google | 8B | 131K | $0.02 / $0.10 | ||
| 44 | Alibaba Cloud / Qwen Team | 4B | — | — | ||
| 45 | Mistral AI | 675B | — | — | ||
| 46 |  Microsoft | 60B | — | — | ||
| 47 | Google | 5B | — | — | ||
| 48 | OpenAI | — | 1.0M | $0.10 / $0.40 | ||
| 49 | Alibaba Cloud / Qwen Team | 2B | — | — | ||
| 50 | Microsoft | 4B | — | — |
What is MMMLU?
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
MMMLU is a text benchmark evaluating models on language, math, reasoning, and general tasks. LLM Stats tracks 51 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 0.9.
Compare leaders on the best AI for language, best AI for math, best AI for reasoning and best AI for general leaderboards.
Current leaders
Claude Mythos Preview from Anthropic currently leads the MMMLU leaderboard with a score of 0.927 across 51 evaluated AI models.
Source paper
- Title
- Measuring Massive Multitask Language Understanding
- Authors
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, and 3 others
- Published
- arXiv
- 2009.03300
Abstract
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
FAQ
Common questions about the MMMLU benchmark and leaderboard.