MLVU
A comprehensive benchmark for multi-task long video understanding that evaluates multimodal large language models on videos ranging from 3 minutes to 2 hours across 9 distinct tasks including reasoning, captioning, recognition, and summarization.
Qwen3.5-122B-A10B from Alibaba Cloud / Qwen Team currently leads the MLVU leaderboard with a score of 0.873 across 9 evaluated AI models.
What MLVU measures
MLVU is a multimodal benchmark that evaluates large language models on long context, multimodal, video, and vision tasks. LLM Stats tracks 9 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.8, with the leader reaching 0.9.
Compare leaders on the best AI for long context, best AI for multimodal, best AI for video and best AI for vision leaderboards.
Publication
- Paper
- MLVU: Benchmarking Multi-task Long Video Understanding
- Authors
- Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, and 8 others
- Published
- arXiv
- 2406.04264
Abstract
The evaluation of Long Video Understanding (LVU) performance poses an important but challenging research problem. Despite previous efforts, the existing video understanding benchmarks are severely constrained by several issues, especially the insufficient lengths of videos, a lack of diversity in video types and evaluation tasks, and the inappropriateness for evaluating LVU performances. To address the above problems, we propose a new benchmark called MLVU (Multi-task Long Video Understanding Benchmark) for the comprehensive and in-depth evaluation of LVU. MLVU presents the following critical values: \textit{1)} The substantial and flexible extension of video lengths, which enables the benchmark to evaluate LVU performance across a wide range of durations. \textit{2)} The inclusion of various video genres, e.g., movies, surveillance footage, egocentric videos, cartoons, game videos, etc., which reflects the models' LVU performances in different scenarios. \textit{3)} The development of diversified evaluation tasks, which enables a comprehensive examination of MLLMs' key abilities in long-video understanding. The empirical study with 23 latest MLLMs reveals significant room for improvement in today's technique, as all existing methods struggle with most of the evaluation tasks and exhibit severe performance degradation when handling longer videos. Additionally, it suggests that factors such as context length, image-understanding ability, and the choice of LLM backbone can play critical roles in future advancements. We anticipate that MLVU will advance the research of long video understanding by providing a comprehensive and in-depth analysis of MLLMs.
Qwen3.5-122B-A10B leads with 87.3%, followed by Qwen3.6 Plus at 86.7% and Qwen3.6-27B at 86.6%.
Progress Over Time
Interactive timeline showing model performance evolution on MLVU
MLVU Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Alibaba Cloud / Qwen Team | 122B | 262K | $0.40 / $3.20 | ||
| 2 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 3 | Alibaba Cloud / Qwen Team | 28B | 262K | $0.60 / $3.60 | ||
| 4 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 5 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 6 | Alibaba Cloud / Qwen Team | 35B | 262K | $0.25 / $2.00 | ||
| 7 | Alibaba Cloud / Qwen Team | 236B | 262K | $0.30 / $1.50 | ||
| 8 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 9 | Alibaba Cloud / Qwen Team | 8B | — | — |
FAQ
Common questions about MLVU.
More evaluations to explore
Related benchmarks in the same category
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions
MMMU (Massive Multi-discipline Multimodal Understanding) is a benchmark designed to evaluate multimodal models on college-level subject knowledge and deliberate reasoning. Contains 11.5K meticulously collected multimodal questions from college exams, quizzes, and textbooks, covering six core disciplines: Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering across 30 subjects and 183 subfields.
A more robust multi-discipline multimodal understanding benchmark that enhances MMMU through a three-step process: filtering text-only answerable questions, augmenting candidate options, and introducing vision-only input settings. Achieves significantly lower model performance (16.8-26.9%) compared to original MMMU, providing more rigorous evaluation that closely mimics real-world scenarios.
CharXiv-R is the reasoning component of the CharXiv benchmark, focusing on complex reasoning questions that require synthesizing information across visual chart elements. It evaluates multimodal large language models on their ability to understand and reason about scientific charts from arXiv papers through various reasoning tasks.
MathVista evaluates mathematical reasoning of foundation models in visual contexts. It consists of 6,141 examples derived from 28 existing multimodal datasets and 3 newly created datasets (IQTest, FunctionQA, and PaperQA), combining challenges from diverse mathematical and visual tasks to assess models' ability to understand complex figures and perform rigorous reasoning.