- Organizations
- MMMU Team
MMMU Team: AI benchmarks & research
Academic team behind MMMU (Ohio State, CMU, Waterloo and others). Maintains MMMU, MMMU-Pro and Video-MMMU.
Maintained by MMMU Team
7 benchmarks- MMMU
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
- Leader
Qwen3.6 Plus86.0%- Models scored
- 66
- MMMU-Pro
A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings. Achieves significantly lower model performance (16.8-26.9%) compared to original MMMU, providing more rigorous evaluation that closely mimics real-world scenarios.
- Leader
GPT-6.1 Sol86.0%
- Models scored
- 77
- MMMU-Pro (with tools)
MMMU-Pro variant evaluated with tool access enabled.
- Leader
Kimi K3.1 Pro90.0%
- Models scored
- 5
- MMMUval
Validation set for MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning) benchmark, designed to evaluate multimodal models on massive multi-discipline tasks demanding college-level subject knowledge and deliberate reasoning across Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering.
- Leader
Qwen3 VL 235B A22B Thinking80.6%- Models scored
- 4
- MMMU (val)
Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables.
- Leader
Qwen3 VL 32B Thinking78.1%- Models scored
- 14
- MMMU (validation)
Validation set of the Massive Multi-discipline Multimodal Understanding and Reasoning benchmark. Features college-level multimodal questions across 6 core disciplines (Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, Tech & Engineering) spanning 30 subjects and 183 subfields with diverse image types including charts, diagrams, maps, and tables.
- Leader
Claude Opus 4.580.7%
- Models scored
- 4
- VideoMMMU
Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.
- Leader
Gemini 3 Pro87.6%
- Models scored
- 29