ARC-C
Progress Over Time
Interactive timeline showing model performance evolution on ARC-C
ARC-C Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Xiaomi | 1.0T | 1.0M | $0.43 / $0.87 | ||
| 2 | 405B | — | — | |||
| 3 |  Anthropic | — | — | — | ||
| 4 |  Amazon | — | — | — | ||
| 4 | 70B | — | — | |||
| 6 | Anthropic | — | — | — | ||
| 7 |  AI21 Labs | 398B | — | — | ||
| 8 | Amazon | — | — | — | ||
| 9 |  Mistral AI | 24B | — | — | ||
| 10 |  Microsoft | 60B | — | — | ||
| 11 | Amazon | — | — | — | ||
| 12 | Anthropic | — | — | — | ||
| 13 | AI21 Labs | 52B | — | — | ||
| 14 | Microsoft | 4B | — | — | ||
| 15 | Microsoft | 4B | — | — | ||
| 16 | 8B | — | — | |||
| 17 | 3B | — | — | |||
| 18 | Mistral AI | 8B | — | — | ||
| 19 |  Google | 27B | — | — | ||
| 20 |  Cohere | 104B | — | — | ||
| 21 |  Alibaba Cloud / Qwen Team | 32B | — | — | ||
| 22 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 23 | 70B | — | — | |||
| 24 | Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 25 | Google | 9B | — | — | ||
| 26 | Alibaba Cloud / Qwen Team | 15B | — | — | ||
| 27 |  Nous Research | 70B | — | — | ||
| 28 | 2B | — | — | |||
| 28 | Google | 8B | — | — | ||
| 30 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 31 | 2B | — | — | |||
| 31 | Google | 8B | — | — | ||
| 33 | 8B | — | — | |||
| 34 |  Baidu | 21B | — | — |
What is ARC-C?
The AI2 Reasoning Challenge (ARC) Challenge Set is a multiple-choice question-answering benchmark containing grade-school level science questions that require advanced reasoning capabilities. ARC-C specifically contains questions that were answered incorrectly by both retrieval-based and word co-occurrence algorithms, making it a particularly challenging subset designed to test commonsense reasoning abilities in AI systems.
ARC-C is a text benchmark evaluating models on reasoning and general tasks. LLM Stats tracks 34 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 1.0.
Compare leaders on the best AI for reasoning and best AI for general leaderboards.
Current leaders
MiMo-V2.5-Pro from Xiaomi currently leads the ARC-C leaderboard with a score of 0.972 across 34 evaluated AI models.
Source paper
- Title
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- Authors
- Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, and 3 others
- Published
- arXiv
- 1803.05457
Abstract
We present a new question set, text corpus, and baselines assembled to encourage AI research in advanced question answering. Together, these constitute the AI2 Reasoning Challenge (ARC), which requires far more powerful knowledge and reasoning than previous challenges such as SQuAD or SNLI. The ARC question set is partitioned into a Challenge Set and an Easy Set, where the Challenge Set contains only questions answered incorrectly by both a retrieval-based algorithm and a word co-occurence algorithm. The dataset contains only natural, grade-school science questions (authored for human tests), and is the largest public-domain set of this kind (7,787 questions). We test several baselines on the Challenge Set, including leading neural models from the SQuAD and SNLI tasks, and find that none are able to significantly outperform a random baseline, reflecting the difficult nature of this task. We are also releasing the ARC Corpus, a corpus of 14M science sentences relevant to the task, and implementations of the three neural baseline models tested. Can your model perform better? We pose ARC as a challenge to the community.
FAQ
Common questions about the ARC-C benchmark and leaderboard.