MMMU
Progress Over Time
Interactive timeline showing model performance evolution on MMMU
MMMU Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 2 |  OpenAI | — | — | — | ||
| 2 | OpenAI | — | 400K | $1.25 / $10.00 | ||
| 2 | OpenAI | — | 400K | $1.25 / $10.00 | ||
| 5 | OpenAI | — | — | — | ||
| 6 | Alibaba Cloud / Qwen Team | 122B | — | — | ||
| 7 | OpenAI | — | — | — | ||
| 7 | Alibaba Cloud / Qwen Team | 28B | 262K | $0.60 / $3.60 | ||
| 9 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 10 | — | — | — | |||
| 11 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 12 | OpenAI | — | — | — | ||
| 13 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 14 |  Google | — | 1.0M | $0.30 / $2.50 | ||
| 15 | Google | — | 1.0M | $1.25 / $10.00 | ||
| 16 |  StepFun | 10B | — | — | ||
| 17 |  xAI | — | 128K | $3.00 / $15.00 | ||
| 18 | OpenAI | — | — | — | ||
| 19 | — | — | — | |||
| 20 | OpenAI | — | — | — | ||
| 21 |  Cohere | 218B | — | — | ||
| 22 |  Anthropic | — | — | — | ||
| 23 | OpenAI | — | 1.0M | $2.00 / $8.00 | ||
| 24 | Anthropic | — | — | — | ||
| 25 |  Meta | 400B | — | — | ||
| 26 | Google | — | — | — | ||
| 27 | OpenAI | — | 1.0M | $0.40 / $1.60 | ||
| 28 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 29 | Google | — | — | — | ||
| 30 | Alibaba Cloud / Qwen Team | 73B | — | — | ||
| 31 | Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 32 | Alibaba Cloud / Qwen Team | 34B | — | — | ||
| 32 |  Moonshot AI | — | — | — | ||
| 34 | Meta | 109B | — | — | ||
| 35 | Anthropic | — | — | — | ||
| 36 | Google | — | — | — | ||
| 37 | xAI | — | — | — | ||
| 38 | Google | — | — | — | ||
| 39 |  Mistral AI | 124B | — | — | ||
| 40 | xAI | — | — | — | ||
| 41 | Mistral AI | 24B | — | — | ||
| 42 | Google | — | — | — | ||
| 43 |  Amazon | — | — | — | ||
| 44 | 90B | — | — | |||
| 45 | OpenAI | — | — | — | ||
| 46 | Mistral AI | 24B | — | — | ||
| 46 | Mistral AI | 24B | — | — | ||
| 48 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 49 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 50 | Amazon | — | — | — |
What is MMMU?
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
MMMU is a multimodal benchmark evaluating models on multimodal, reasoning, general, healthcare, and vision tasks. LLM Stats tracks 63 models on this benchmark, scored on a 0–1 scale. The current average is 0.7, with the leader at 0.9.
Compare leaders on the best AI for multimodal, best AI for reasoning, best AI for general, best AI for healthcare and best AI for vision leaderboards.
Current leaders
Qwen3.6 Plus from Alibaba Cloud / Qwen Team currently leads the MMMU leaderboard with a score of 0.860 across 63 evaluated AI models.
Source paper
- Title
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- Authors
- Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, and 18 others
- Published
- arXiv
- 2311.16502
Abstract
We introduce MMMU: a new benchmark designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning. MMMU includes 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering. These questions span 30 subjects and 183 subfields, comprising 30 highly heterogeneous image types, such as charts, diagrams, maps, tables, music sheets, and chemical structures. Unlike existing benchmarks, MMMU focuses on advanced perception and reasoning with domain-specific knowledge, challenging models to perform tasks akin to those faced by experts. The evaluation of 14 open-source LMMs as well as the proprietary GPT-4V(ision) and Gemini highlights the substantial challenges posed by MMMU. Even the advanced GPT-4V and Gemini Ultra only achieve accuracies of 56% and 59% respectively, indicating significant room for improvement. We believe MMMU will stimulate the community to build next-generation multimodal foundation models towards expert artificial general intelligence.
FAQ
Common questions about the MMMU benchmark and leaderboard.