Arena-Hard v2

Arena-Hard-Auto v2 is a challenging benchmark consisting of 500 carefully curated prompts sourced from Chatbot Arena and WildChat-1M, designed to evaluate large language models on real-world user queries. The benchmark covers diverse domains including open-ended software engineering problems, mathematics, creative writing, and technical problem-solving. It uses LLM-as-a-Judge for automatic evaluation, achieving 98.6% correlation with human preference rankings while providing 3x higher separation of model performances compared to MT-Bench. The benchmark emphasizes prompt specificity, complexity, and domain knowledge to better distinguish between model capabilities.

Arena-Hard v2 Leaderboard

AllOpenProprietary

16 models

#	Model	Score	Size	Context	Cost	License
1	MiMo-V2-Flash Xiaomi	0.862	309B	256K	$0.10 / $0.30
2	Qwen3-Next-80B-A3B-Instruct Alibaba Cloud / Qwen Team	0.827	80B	66K	$0.15 / $1.50
3	Qwen3-235B-A22B-Thinking-2507 Alibaba Cloud / Qwen Team	0.797	235B	262K	$0.30 / $3.00
4	Qwen3-235B-A22B-Instruct-2507 Alibaba Cloud / Qwen Team	0.792	235B	262K	$0.15 / $0.80
5	Qwen3 VL 235B A22B Instruct Alibaba Cloud / Qwen Team	0.774	236B	262K	$0.30 / $1.49
6	Nemotron 3 Super (120B A12B) NVIDIA	0.739	120B	—	—
7	Sarvam-105B Sarvam AI	0.710	105B	—	—
8	Nemotron 3 Nano (30B A3B) NVIDIA	0.677	32B	262K	$0.06 / $0.24
9	Qwen3 VL 32B Instruct Alibaba Cloud / Qwen Team	0.647	33B	—	—
10	Qwen3-Next-80B-A3B-Thinking Alibaba Cloud / Qwen Team	0.623	80B	66K	$0.15 / $1.50
11	Qwen3 VL 32B Thinking Alibaba Cloud / Qwen Team	0.605	33B	—	—
12	Qwen3 VL 30B A3B Instruct Alibaba Cloud / Qwen Team	0.585	31B	262K	$0.20 / $0.70
13	Qwen3 VL 30B A3B Thinking Alibaba Cloud / Qwen Team	0.567	31B	262K	$0.20 / $1.00
14	Qwen3 VL 8B Thinking Alibaba Cloud / Qwen Team	0.511	9B	262K	$0.18 / $2.09
15	Sarvam-30B Sarvam AI	0.490	30B	—	—
16	Qwen3 VL 4B Thinking Alibaba Cloud / Qwen Team	0.368	4B	262K	$0.10 / $1.00