MMLU-Redux
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
Qwen3.7 Max from Alibaba Cloud / Qwen Team currently leads the MMLU-Redux leaderboard with a score of 0.950 across 47 evaluated AI models.
What MMLU-Redux measures
MMLU-Redux is a text benchmark that evaluates large language models on language, math, reasoning, and general tasks. LLM Stats tracks 47 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.9, with the leader reaching 0.9.
Compare leaders on the best AI for language, best AI for math, best AI for reasoning and best AI for general leaderboards.
Publication
- Paper
- Are We Done with MMLU?
- Authors
- Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, and 12 others
- Published
- arXiv
- 2406.04127
Abstract
Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.
Qwen3.7 Max leads with 95.0%, followed by Qwen3.5-397B-A17B at 94.9% and Qwen3.6 Plus at 94.5%.
Progress Over Time
Interactive timeline showing model performance evolution on MMLU-Redux
MMLU-Redux Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Alibaba Cloud / Qwen Team | — | 1.0M | $1.25 / $3.75 | ||
| 2 | Alibaba Cloud / Qwen Team | 397B | 262K | $0.60 / $3.60 | ||
| 3 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 4 |  Moonshot AI | 1.0T | — | — | ||
| 5 | Alibaba Cloud / Qwen Team | 122B | 262K | $0.40 / $3.20 | ||
| 6 | Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 7 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 8 | Alibaba Cloud / Qwen Team | 28B | 262K | $0.60 / $3.60 | ||
| 9 |  DeepSeek | 671B | 131K | $0.55 / $2.19 | ||
| 10 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 10 | Alibaba Cloud / Qwen Team | 35B | 262K | $0.25 / $2.00 | ||
| 12 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 13 | Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 14 |  Xiaomi | 1.0T | 1.0M | $0.43 / $0.87 | ||
| 15 | Moonshot AI | 1.0T | — | — | ||
| 15 | Moonshot AI | 1.0T | — | — | ||
| 17 | Alibaba Cloud / Qwen Team | 80B | — | — | ||
| 18 | Alibaba Cloud / Qwen Team | 236B | 262K | $0.30 / $1.50 | ||
| 19 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 20 | DeepSeek | 671B | — | — | ||
| 21 | Alibaba Cloud / Qwen Team | 9B | — | — | ||
| 22 | Alibaba Cloud / Qwen Team | 80B | — | — | ||
| 22 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 24 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 25 |  Meituan | 560B | — | — | ||
| 26 | DeepSeek | 671B | — | — | ||
| 27 | Alibaba Cloud / Qwen Team | 4B | — | — | ||
| 27 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.18 / $2.09 | ||
| 29 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 30 | Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 31 | Alibaba Cloud / Qwen Team | 73B | — | — | ||
| 32 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $1.00 | ||
| 33 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.08 / $0.50 | ||
| 34 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 35 |  Mistral AI | 675B | — | — | ||
| 35 | Mistral AI | 14B | — | — | ||
| 37 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $0.60 | ||
| 38 | Alibaba Cloud / Qwen Team | 15B | — | — | ||
| 39 | Alibaba Cloud / Qwen Team | 2B | — | — | ||
| 40 | Mistral AI | 8B | — | — | ||
| 41 | Alibaba Cloud / Qwen Team | 32B | — | — | ||
| 42 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 43 | Mistral AI | 3B | — | — | ||
| 44 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 45 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 46 | Alibaba Cloud / Qwen Team | 800M | — | — | ||
| 47 |  Baidu | 21B | — | — |
FAQ
Common questions about MMLU-Redux.
More evaluations to explore
Related benchmarks in the same category
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions