Natural Questions
Progress Over Time
Interactive timeline showing model performance evolution on Natural Questions
Natural Questions Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Google | 27B | — | — | ||
| 2 |  Mistral AI | 12B | — | — | ||
| 3 | Google | 9B | — | — | ||
| 4 | 2B | — | — | |||
| 4 | Google | 8B | — | — | ||
| 6 | 2B | — | — | |||
| 6 | Google | 8B | — | — |
What is Natural Questions?
Natural Questions is a question answering dataset featuring real anonymized queries issued to Google search engine. It contains 307,373 training examples where annotators provide long answers (passages) and short answers (entities) from Wikipedia pages, or mark them as unanswerable.
Natural Questions is a text benchmark evaluating models on reasoning, search, and general tasks. LLM Stats tracks 7 models on this benchmark, scored on a 0–1 scale. The current average is 0.2, with the leader at 0.3.
Compare leaders on the best AI for reasoning, best AI for search and best AI for general leaderboards.
Current leaders
Gemma 2 27B from Google currently leads the Natural Questions leaderboard with a score of 0.345 across 7 evaluated AI models.
Source paper
- Title
- A BERT Baseline for the Natural Questions
- Authors
- Chris Alberti, Kenton Lee, Michael Collins
- Published
- arXiv
- 1901.08634
Abstract
This technical note describes a new baseline for the Natural Questions. Our model is based on BERT and reduces the gap between the model F1 scores reported in the original dataset paper and the human upper bound by 30% and 50% relative for the long and short answer tasks respectively. This baseline has been submitted to the official NQ leaderboard at ai.google.com/research/NaturalQuestions. Code, preprocessed data and pretrained model are available at https://github.com/google-research/language/tree/master/language/question_answering/bert_joint.
FAQ
Common questions about the Natural Questions benchmark and leaderboard.