SimpleQA

SimpleQA is a factuality benchmark developed by OpenAI that measures the short-form factual accuracy of large language models. The benchmark contains 4,326 short, fact-seeking questions that are adversarially collected and designed to have single, indisputable answers. Questions cover diverse topics from science and technology to entertainment, and the benchmark also measures model calibration by evaluating whether models know what they know.

SimpleQA Leaderboard

AllOpenProprietary

44 models

#	Model	Score	Size	Context	Cost	License
1	DeepSeek-V3.2-Exp DeepSeek	0.971	685B	—	—
2	Grok 4 Fast xAI	0.950	—	2.0M	$0.20 / $0.50
3	DeepSeek-V3.1 DeepSeek	0.934	671B	164K	$0.27 / $1.00
4	DeepSeek-R1-0528 DeepSeek	0.923	671B	131K	$0.50 / $2.15
5	ERNIE 5.0 Baidu	0.750	—	—	—
6	Gemini 3 Pro Google	0.721	—	—	—
7	Gemini 3 Flash Google	0.687	—	1.0M	$0.50 / $3.00
8	GPT-4.5 OpenAI	0.625	—	128K	$75.00 / $150.00
9	Qwen3 VL 32B Thinking Alibaba Cloud / Qwen Team	0.554	33B	—	—
10	Qwen3-235B-A22B-Instruct-2507 Alibaba Cloud / Qwen Team	0.543	235B	262K	$0.15 / $0.80
11	Gemini 2.5 Pro Preview 06-05 Google	0.540	—	1.0M	$1.25 / $10.00
12	Qwen3 VL 235B A22B Instruct Alibaba Cloud / Qwen Team	0.519	236B	262K	$0.30 / $1.49
13	Gemini 2.5 Pro Google	0.508	—	1.0M	$1.25 / $10.00
14	Qwen3 VL 8B Thinking Alibaba Cloud / Qwen Team	0.496	9B	262K	$0.18 / $2.09
15	Qwen3 VL 4B Instruct Alibaba Cloud / Qwen Team	0.480	4B	262K	$0.10 / $0.60
16	o1 OpenAI	0.470	—	200K	$15.00 / $60.00
17	Qwen3 VL 235B A22B Thinking Alibaba Cloud / Qwen Team	0.444	236B	262K	$0.45 / $3.49
18	Gemini 3.1 Flash-Lite Google	0.433	—	1.0M	$0.25 / $1.50
19	o1-preview OpenAI	0.424	—	128K	$15.00 / $60.00
20	GPT-4o OpenAI	0.382	—	128K	$2.50 / $10.00
21	Kimi K2 Base Moonshot AI	0.353	1.0T	—	—
22	Kimi K2-Instruct-0905 Moonshot AI	0.310	1.0T	—	—
22	Kimi K2 Instruct Moonshot AI	0.310	1.0T	200K	$0.50 / $0.50
24	Qwen3 VL 30B A3B Instruct Alibaba Cloud / Qwen Team	0.270	31B	262K	$0.20 / $0.70
25	Gemini 2.5 Flash Google	0.269	—	1.0M	$0.30 / $2.50
26	DeepSeek-V3 DeepSeek	0.249	671B	131K	$0.27 / $1.10
27	Qwen3 VL 30B A3B Thinking Alibaba Cloud / Qwen Team	0.239	31B	262K	$0.20 / $1.00
28	Mistral Large 3 (675B Instruct 2512 NVFP4) Mistral AI	0.238	675B	—	—
28	Mistral Large 3 (675B Instruct 2512) Mistral AI	0.238	675B	262K	$0.50 / $1.50
28	Mistral Large 3 (675B Base) Mistral AI	0.238	675B	—	—
28	Mistral Large 3 (675B Instruct 2512 Eagle) Mistral AI	0.238	675B	—	—
32	Gemini 2.0 Flash-Lite Google	0.217	—	1.0M	$0.07 / $0.30
33	MiniMax M1 80K MiniMax	0.185	456B	1.0M	$0.55 / $2.20
34	MiniMax M1 40K MiniMax	0.179	456B	—	—
35	o3-mini OpenAI	0.150	—	200K	$1.10 / $4.40
36	Mistral Small 3.2 24B Instruct Mistral AI	0.121	24B	—	—
37	Gemini 2.5 Flash-Lite Google	0.107	—	1.0M	$0.10 / $0.40
38	Mistral Small 3.1 24B Instruct Mistral AI	0.104	24B	—	—
39	Gemma 3 27B Google	0.100	27B	131K	$0.10 / $0.20
40	Gemma 3 12B Google	0.063	12B	131K	$0.05 / $0.10
41	Gemma 3 4B Google	0.040	4B	131K	$0.02 / $0.04
42	Phi 4 Microsoft	0.030	15B	16K	$0.07 / $0.14
43	Gemma 3 1B Google	0.022	1B	—	—
44	ERNIE 4.5 Baidu	0.018	21B	128K	$0.40 / $4.00