PopQA
Progress Over Time
Interactive timeline showing model performance evolution on PopQA
PopQA Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | 8B | — | — | |||
| 1 | 8B | — | — | |||
| 3 | 7B | — | — |
What is PopQA?
PopQA is an entity-centric open-domain question-answering dataset consisting of 14,000 QA pairs designed to evaluate language models' ability to memorize and recall factual knowledge across entities with varying popularity levels. The dataset probes both parametric memory (stored in model parameters) and non-parametric memory effectiveness, with questions covering 16 diverse relationship types from Wikidata converted to natural language using templates. Created by sampling knowledge triples from Wikidata and converting them to natural language questions, focusing on long-tail entities to understand LMs' strengths and limitations in memorizing factual knowledge.
PopQA is a text benchmark evaluating models on reasoning and general tasks. LLM Stats tracks 3 models on this benchmark, scored on a 0–1 scale. The current average is 0.3, with the leader at 0.3.
Compare leaders on the best AI for reasoning and best AI for general leaderboards.
Current leaders
Granite 3.3 8B Base from IBM currently leads the PopQA leaderboard with a score of 0.262 across 3 evaluated AI models.
Source paper
- Title
- When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
- Authors
- Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, and 2 others
- Published
- arXiv
- 2212.10511
Abstract
Despite their impressive performance on diverse tasks, large language models (LMs) still struggle with tasks requiring rich world knowledge, implying the limitations of relying solely on their parameters to encode a wealth of world knowledge. This paper aims to understand LMs' strengths and limitations in memorizing factual knowledge, by conducting large-scale knowledge probing experiments of 10 models and 4 augmentation methods on PopQA, our new open-domain QA dataset with 14k questions. We find that LMs struggle with less popular factual knowledge, and that scaling fails to appreciably improve memorization of factual knowledge in the long tail. We then show that retrieval-augmented LMs largely outperform orders of magnitude larger LMs, while unassisted LMs remain competitive in questions about high-popularity entities. Based on those findings, we devise a simple, yet effective, method for powerful and efficient retrieval-augmented LMs, which retrieves non-parametric memories only when necessary. Experimental results show that this significantly improves models' performance while reducing the inference costs.
FAQ
Common questions about the PopQA benchmark and leaderboard.