MMLU-redux-2.0
Progress Over Time
Interactive timeline showing model performance evolution on MMLU-redux-2.0
MMLU-redux-2.0 Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Moonshot AI | 1.0T | — | — |
What is MMLU-redux-2.0?
A curated version of the MMLU benchmark featuring manually re-annotated 5,700 questions across 57 subjects to identify and correct errors in the original dataset. Addresses the 6.49% error rate found in MMLU and provides more reliable evaluation metrics for language models.
MMLU-redux-2.0 is a text benchmark evaluating models on language, math, reasoning, and general tasks. LLM Stats tracks 1 models on this benchmark, scored on a 0–1 scale. The current average is 0.9, with the leader at 0.9.
Compare leaders on the best AI for language, best AI for math, best AI for reasoning and best AI for general leaderboards.
Current leaders
Kimi K2 Base from Moonshot AI currently leads the MMLU-redux-2.0 leaderboard with a score of 0.902 across 1 evaluated AI models.
Source paper
- Title
- Are We Done with MMLU?
- Authors
- Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, and 12 others
- Published
- arXiv
- 2406.04127
Abstract
Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0.
FAQ
Common questions about the MMLU-redux-2.0 benchmark and leaderboard.