MMMU-Pro (with tools)
Progress Over Time
Interactive timeline showing model performance evolution on MMMU-Pro (with tools)
MMMU-Pro (with tools) Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  GPT-5.6 SolNew OpenAI | — | 1.1M | $5.00 / $30.00 | ||
| 2 | OpenAI | — | 1.1M | $2.50 / $15.00 | ||
| 3 | GPT-5.6 LunaNew OpenAI | — | 1.1M | $1.00 / $6.00 |
What is MMMU-Pro (with tools)?
MMMU-Pro variant evaluated with tool access enabled.
MMMU-Pro (with tools) is a multimodal benchmark evaluating models on multimodal, reasoning, general, and vision tasks. LLM Stats tracks 3 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 0.8.
Compare leaders on the best AI for multimodal, best AI for reasoning, best AI for general and best AI for vision leaderboards.
Current leaders
GPT-5.6 Sol from OpenAI currently leads the MMMU-Pro (with tools) leaderboard with a score of 0.846 across 3 evaluated AI models.
Source paper
- Title
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
- Authors
- Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, and 9 others
- Published
- arXiv
- 2409.02813
Abstract
This paper introduces MMMU-Pro, a robust version of the Massive Multi-discipline Multimodal Understanding and Reasoning (MMMU) benchmark. MMMU-Pro rigorously assesses multimodal models' true understanding and reasoning capabilities through a three-step process based on MMMU: (1) filtering out questions answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. This setting challenges AI to truly "see" and "read" simultaneously, testing a fundamental human cognitive skill of seamlessly integrating visual and textual information. Results show that model performance is substantially lower on MMMU-Pro than on MMMU, ranging from 16.8% to 26.9% across models. We explore the impact of OCR prompts and Chain of Thought (CoT) reasoning, finding that OCR prompts have minimal effect while CoT generally improves performance. MMMU-Pro provides a more rigorous evaluation tool, closely mimicking real-world scenarios and offering valuable directions for future research in multimodal AI.
FAQ
Common questions about the MMMU-Pro (with tools) benchmark and leaderboard.