MathVision
Progress Over Time
Interactive timeline showing model performance evolution on MathVision
MathVision Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  ByteDance | — | — | — | ||
| 2 |  Moonshot AI | 1.0T | 262K | $0.75 / $3.50 | ||
| 3 | ByteDance | — | — | — | ||
| 4 |  Alibaba Cloud / Qwen Team | — | 1.0M | $0.32 / $1.28 | ||
| 5 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 6 | Alibaba Cloud / Qwen Team | 122B | — | — | ||
| 7 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 8 |  Google | 31B | 262K | $0.13 / $0.38 | ||
| 9 | Moonshot AI | 1.0T | — | — | ||
| 10 | Alibaba Cloud / Qwen Team | 35B | — | — | ||
| 11 | Google | 25B | 262K | $0.13 / $0.40 | ||
| 12 | Google | 12B | — | — | ||
| 13 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 14 |  StepFun | 10B | — | — | ||
| 15 | Google | 25B | — | — | ||
| 16 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 17 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 18 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 19 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 20 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.18 / $2.09 | ||
| 21 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 22 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $1.00 | ||
| 23 | Google | 8B | — | — | ||
| 24 | Alibaba Cloud / Qwen Team | 9B | — | — | ||
| 25 | Google | 5B | — | — | ||
| 26 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $0.60 | ||
| 27 | Alibaba Cloud / Qwen Team | 34B | — | — | ||
| 28 | Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 29 | Alibaba Cloud / Qwen Team | 73B | — | — | ||
| 30 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 31 | Alibaba Cloud / Qwen Team | 7B | — | — |
What is MathVision?
MATH-Vision is a dataset designed to measure multimodal mathematical reasoning capabilities. It focuses on evaluating how well models can solve mathematical problems that require both visual understanding and mathematical reasoning, bridging the gap between visual and mathematical domains.
MathVision is a multimodal benchmark evaluating models on math, multimodal, and vision tasks. LLM Stats tracks 31 models on this benchmark, scored on a 0–1 scale. The current average is 0.7, with the leader at 0.9.
Compare leaders on the best AI for math, best AI for multimodal and best AI for vision leaderboards.
Current leaders
Seed 2.1 Pro from ByteDance currently leads the MathVision leaderboard with a score of 0.945 across 31 evaluated AI models.
Source paper
- Title
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Authors
- Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, and 2 others
- Published
- arXiv
- 2402.14804
Abstract
Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io
FAQ
Common questions about the MathVision benchmark and leaderboard.