SimpleQA
Progress Over Time
Interactive timeline showing model performance evolution on SimpleQA
SimpleQA Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  DeepSeek | 685B | — | — | ||
| 2 |  xAI | — | 2.0M | $0.20 / $0.50 | ||
| 3 | DeepSeek | 671B | — | — | ||
| 4 | DeepSeek | 671B | 131K | $0.55 / $2.19 | ||
| 5 |  Baidu | — | — | — | ||
| 6 |  Google | — | — | — | ||
| 7 | Google | — | 1.0M | $0.50 / $3.00 | ||
| 8 |  OpenAI | — | — | — | ||
| 9 | DeepSeek | 1.6T | 1.0M | $1.60 / $3.20 | ||
| 10 |  Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 11 | Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 12 | — | — | — | |||
| 13 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 14 | Google | — | 1.0M | $1.25 / $10.00 | ||
| 15 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.18 / $2.09 | ||
| 16 | Alibaba Cloud / Qwen Team | 4B | 262K | $0.10 / $0.60 | ||
| 17 | OpenAI | — | — | — | ||
| 18 | Alibaba Cloud / Qwen Team | 236B | — | — | ||
| 19 | Google | — | 1.0M | $0.25 / $1.50 | ||
| 20 | OpenAI | — | — | — | ||
| 21 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 22 |  Moonshot AI | 1.0T | — | — | ||
| 23 | DeepSeek | 284B | 1.0M | $0.10 / $0.20 | ||
| 24 | Moonshot AI | 1.0T | — | — | ||
| 24 | Moonshot AI | 1.0T | — | — | ||
| 26 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 27 | Google | — | 1.0M | $0.30 / $2.50 | ||
| 28 | DeepSeek | 671B | — | — | ||
| 29 | Alibaba Cloud / Qwen Team | 31B | — | — | ||
| 30 | 675B | — | — | |||
| 30 | 675B | — | — | |||
| 30 |  Mistral AI | 675B | — | — | ||
| 30 | Mistral AI | 675B | 262K | $0.50 / $1.50 | ||
| 34 | Google | — | — | — | ||
| 35 |  MiniMax | 456B | — | — | ||
| 36 | MiniMax | 456B | — | — | ||
| 37 | OpenAI | — | — | — | ||
| 38 | Mistral AI | 24B | — | — | ||
| 39 | Google | — | — | — | ||
| 40 | Mistral AI | 24B | — | — | ||
| 41 | Google | 27B | — | — | ||
| 42 | Google | 12B | — | — | ||
| 43 | Google | 4B | — | — | ||
| 44 |  Microsoft | 15B | — | — | ||
| 45 | Google | 1B | — | — | ||
| 46 | Baidu | 21B | — | — |
Sub-benchmarks
What is SimpleQA?
SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.
SimpleQA is a text benchmark evaluating models on reasoning, factuality, and general tasks. LLM Stats tracks 46 models on this benchmark, scored on a 0–1 scale. The current average is 0.4, with the leader at 1.0.
Compare leaders on the best AI for reasoning, best AI for factuality and best AI for general leaderboards.
Current leaders
DeepSeek-V3.2-Exp from DeepSeek currently leads the SimpleQA leaderboard with a score of 0.971 across 46 evaluated AI models.
Source paper
- Title
- Measuring short-form factuality in large language models
- Authors
- Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, and 4 others
- Published
- arXiv
- 2411.04368
Abstract
We present SimpleQA, a benchmark that evaluates the ability of language models to answer short, fact-seeking questions. We prioritized two properties in designing this eval. First, SimpleQA is challenging, as it is adversarially collected against GPT-4 responses. Second, responses are easy to grade, because questions are created such that there exists only a single, indisputable answer. Each answer in SimpleQA is graded as either correct, incorrect, or not attempted. A model with ideal behavior would get as many questions correct as possible while not attempting the questions for which it is not confident it knows the correct answer. SimpleQA is a simple, targeted evaluation for whether models "know what they know," and our hope is that this benchmark will remain relevant for the next few generations of frontier models. SimpleQA can be found at https://github.com/openai/simple-evals.
FAQ
Common questions about the SimpleQA benchmark and leaderboard.