BIG-Bench Extra Hard

BIG-Bench Extra Hard (BBEH) is a challenging benchmark that replaces each task in BIG-Bench Hard with a novel task that probes similar reasoning capabilities but exhibits significantly increased difficulty. The benchmark contains 23 tasks testing diverse reasoning skills including many-hop reasoning, causal understanding, spatial reasoning, temporal arithmetic, geometric reasoning, linguistic reasoning, logic puzzles, and humor understanding. Designed to address saturation on existing benchmarks where state-of-the-art models achieve near-perfect scores, BBEH shows substantial room for improvement with best models achieving only 9.8-44.8% average accuracy.

BIG-Bench Extra Hard Leaderboard

AllOpenProprietary

9 models

#	Model	Score	Size	Context	Cost	License
1	Gemma 4 31BNew Google	0.744	31B	—	—
2	Gemma 4 26B-A4BNew Google	0.648	25B	—	—
3	Gemma 4 E4BNew Google	0.331	8B	—	—
4	Gemma 4 E2BNew Google	0.219	5B	—	—
5	Gemma 3 27B Google	0.193	27B	131K	$0.10 / $0.20
6	Gemma 3 12B Google	0.163	12B	131K	$0.05 / $0.10
7	Gemini Diffusion Google	0.150	—	—	—
8	Gemma 3 4B Google	0.110	4B	131K	$0.02 / $0.04
9	Gemma 3 1B Google	0.072	1B	—	—