BBH
Big-Bench Hard (BBH) is a suite of 23 challenging tasks selected from BIG-Bench for which prior language model evaluations did not outperform the average human-rater. These tasks require multi-step reasoning across diverse domains including arithmetic, logical reasoning, reading comprehension, and commonsense reasoning. The benchmark was designed to test capabilities believed to be beyond current language models and focuses on evaluating complex reasoning skills including temporal understanding, spatial reasoning, causal understanding, and deductive logical reasoning.
Qwen3 235B A22B from Alibaba Cloud / Qwen Team currently leads the BBH leaderboard with a score of 0.889 across 12 evaluated AI models.
What BBH measures
BBH is a text benchmark that evaluates large language models on language, math, and reasoning tasks. LLM Stats tracks 12 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.8, with the leader reaching 0.9.
Compare leaders on the best AI for language, best AI for math and best AI for reasoning leaderboards.
Publication
- Paper
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- Authors
- Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, and 7 others
- Published
- arXiv
- 2210.09261
Abstract
BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models? In this work, we focus on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average human-rater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.
Qwen3 235B A22B leads with 88.9%, followed by 
MiMo-V2.5-Pro at 88.4% and 
Nova Pro at 86.9%.
Progress Over Time
Interactive timeline showing model performance evolution on BBH
BBH Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 2 | Xiaomi | 1.0T | 1.0M | $0.43 / $0.87 | ||
| 3 | Amazon | — | — | — | ||
| 4 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 5 |  DeepSeek | 236B | — | — | ||
| 6 | Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 6 | Amazon | — | — | — | ||
| 8 |  OpenBMB | 9B | — | — | ||
| 9 | Amazon | — | — | — | ||
| 10 | Alibaba Cloud / Qwen Team | 15B | — | — | ||
| 11 |  Nous Research | 70B | — | — | ||
| 12 |  Baidu | 21B | — | — |
FAQ
Common questions about BBH.
More evaluations to explore
Related benchmarks in the same category
A challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. Questions are Google-proof and extremely difficult, with PhD experts reaching 65% accuracy.
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
All 30 problems from the 2025 American Invitational Mathematics Examination (AIME I and AIME II), testing olympiad-level mathematical reasoning with integer answers from 000-999. Used as an AI benchmark to evaluate large language models' ability to solve complex mathematical problems requiring multi-step logical deductions and structured symbolic reasoning.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.
Humanity's Last Exam (HLE) is a multi-modal academic benchmark with 2,500 questions across mathematics, humanities, and natural sciences, designed to test LLM capabilities at the frontier of human knowledge with unambiguous, verifiable solutions