VideoMMMU
Progress Over Time
Interactive timeline showing model performance evolution on VideoMMMU
VideoMMMU Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Google | — | — | — | ||
| 2 | Google | — | 1.0M | $0.50 / $3.00 | ||
| 3 |  Moonshot AI | 1.0T | — | — | ||
| 4 |  OpenAI | — | 400K | $1.75 / $14.00 | ||
| 5 |  Alibaba Cloud / Qwen Team | — | 1.0M | $0.32 / $1.28 | ||
| 6 | Google | — | 1.0M | $0.25 / $1.50 | ||
| 7 |  MiniMax | — | 1.0M | $0.30 / $1.20 | ||
| 7 | OpenAI | — | — | — | ||
| 9 | Alibaba Cloud / Qwen Team | 28B | 262K | $0.60 / $3.60 | ||
| 10 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 11 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 12 | — | — | — | |||
| 13 | OpenAI | — | — | — | ||
| 14 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 15 | Alibaba Cloud / Qwen Team | 122B | — | — | ||
| 16 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 17 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 18 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 19 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 20 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 21 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.18 / $2.09 | ||
| 22 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $1.00 | ||
| 23 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 24 | Alibaba Cloud / Qwen Team | 9B | — | — | ||
| 25 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 26 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $0.60 |
What is VideoMMMU?
Video-MMMU evaluates Large Multimodal Models' ability to acquire knowledge from expert-level professional videos across six disciplines through three cognitive stages: perception, comprehension, and adaptation. Contains 300 videos and 900 human-annotated questions spanning Art, Business, Science, Medicine, Humanities, and Engineering.
VideoMMMU is a multimodal benchmark evaluating models on multimodal, reasoning, healthcare, and vision tasks. LLM Stats tracks 26 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 0.9.
Compare leaders on the best AI for multimodal, best AI for reasoning, best AI for healthcare and best AI for vision leaderboards.
Current leaders
Gemini 3 Pro from Google currently leads the VideoMMMU leaderboard with a score of 0.876 across 26 evaluated AI models.
Source paper
- Title
- Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos
- Authors
- Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, and 4 others
- Published
- arXiv
- 2501.13826
Abstract
Humans acquire knowledge through three cognitive stages: perceiving information, comprehending knowledge, and adapting knowledge to solve novel problems. Videos serve as an effective medium for this learning process, facilitating a progression through these cognitive stages. However, existing video benchmarks fail to systematically evaluate the knowledge acquisition capabilities in Large Multimodal Models (LMMs). To address this gap, we introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark designed to assess LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across six disciplines, evaluating knowledge acquisition through stage-aligned question-answer pairs: Perception, Comprehension, and Adaptation. A proposed knowledge gain metric, Δknowledge, quantifies improvement in performance after video viewing. Evaluation of LMMs reveals a steep decline in performance as cognitive demands increase and highlights a significant gap between human and model knowledge acquisition, underscoring the need for methods to enhance LMMs' capability to learn and adapt from videos.
FAQ
Common questions about the VideoMMMU benchmark and leaderboard.