ChartQA
Progress Over Time
Interactive timeline showing model performance evolution on ChartQA
ChartQA Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Anthropic | — | — | — | ||
| 2 |  Meta | 400B | — | — | ||
| 3 |  Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 4 |  Amazon | — | — | — | ||
| 5 | Meta | 109B | — | — | ||
| 6 | Alibaba Cloud / Qwen Team | 73B | — | — | ||
| 7 |  Mistral AI | 124B | — | — | ||
| 8 | Mistral AI | 24B | — | — | ||
| 9 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 10 | Amazon | — | — | — | ||
| 11 |  DeepSeek | 27B | — | — | ||
| 12 |  OpenAI | — | 128K | $2.50 / $10.00 | ||
| 13 | 90B | — | — | |||
| 14 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 15 | DeepSeek | 16B | — | — | ||
| 16 | 11B | — | — | |||
| 17 |  Microsoft | 4B | — | — | ||
| 17 | Mistral AI | 12B | — | — | ||
| 19 | Microsoft | 6B | — | — | ||
| 20 | DeepSeek | 3B | — | — | ||
| 21 |  Google | 27B | — | — | ||
| 22 |  xAI | — | — | — | ||
| 23 | Google | 12B | — | — | ||
| 24 | Google | 4B | — | — |
What is ChartQA?
ChartQA is a large-scale benchmark comprising 9.6K human-written questions and 23.1K questions generated from human-written chart summaries, designed to evaluate models' abilities in visual and logical reasoning over charts.
ChartQA is a multimodal benchmark evaluating models on multimodal, reasoning, and vision tasks. LLM Stats tracks 24 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 0.9.
Compare leaders on the best AI for multimodal, best AI for reasoning and best AI for vision leaderboards.
Current leaders
Claude 3.5 Sonnet from Anthropic currently leads the ChartQA leaderboard with a score of 0.908 across 24 evaluated AI models.
Source paper
- Title
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Authors
- Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and 1 others
- Published
- arXiv
- 2203.10244
Abstract
Charts are very popular for analyzing data. When exploring charts, people often ask a variety of complex reasoning questions that involve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such complex reasoning questions as their questions are template-based and answers come from a fixed-vocabulary. In this work, we present a large-scale benchmark covering 9.6K human-written questions as well as 23.1K questions generated from human-written chart summaries. To address the unique challenges in our benchmark involving visual and logical reasoning over charts, we present two transformer-based models that combine visual features and the data table of the chart in a unified way to answer questions. While our models achieve the state-of-the-art results on the previous datasets as well as on our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions.
FAQ
Common questions about the ChartQA benchmark and leaderboard.