PIQA
Progress Over Time
Interactive timeline showing model performance evolution on PIQA
PIQA Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Microsoft | 60B | — | — | ||
| 2 |  Nous Research | 70B | — | — | ||
| 3 |  Google | 27B | — | — | ||
| 4 | Google | 9B | — | — | ||
| 5 | Microsoft | 4B | — | — | ||
| 5 | Google | 8B | — | — | ||
| 5 | 2B | — | — | |||
| 8 | 2B | — | — | |||
| 8 | Google | 8B | — | — | ||
| 10 | Microsoft | 4B | — | — | ||
| 11 |  Baidu | 21B | — | — |
Sub-benchmarks
What is PIQA?
PIQA (Physical Interaction: Question Answering) is a benchmark dataset for physical commonsense reasoning in natural language. It tests AI systems' ability to answer questions requiring physical world knowledge through multiple choice questions with everyday situations, focusing on atypical solutions inspired by instructables.com. The dataset contains 21,000 multiple choice questions where models must choose the most appropriate solution for physical interactions.
PIQA is a text benchmark evaluating models on physics, reasoning, and general tasks. LLM Stats tracks 11 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 0.9.
Compare leaders on the best AI for physics, best AI for reasoning and best AI for general leaderboards.
Current leaders
Phi-3.5-MoE-instruct from Microsoft currently leads the PIQA leaderboard with a score of 0.886 across 11 evaluated AI models.
Source paper
- Title
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Authors
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and 1 others
- Published
- arXiv
- 1911.11641
Abstract
To apply eyeshadow without a brush, should I use a cotton swab or a toothpick? Questions requiring this kind of physical commonsense pose a challenge to today's natural language understanding systems. While recent pretrained models (such as BERT) have made progress on question answering over more abstract domains - such as news articles and encyclopedia entries, where text is plentiful - in more physical domains, text is inherently limited due to reporting bias. Can AI systems learn to reliably answer physical common-sense questions without experiencing the physical world? In this paper, we introduce the task of physical commonsense reasoning and a corresponding benchmark dataset Physical Interaction: Question Answering or PIQA. Though humans find the dataset easy (95% accuracy), large pretrained models struggle (77%). We provide analysis about the dimensions of knowledge that existing models lack, which offers significant opportunities for future research.
FAQ
Common questions about the PIQA benchmark and leaderboard.