SQuALITY
Progress Over Time
Interactive timeline showing model performance evolution on SQuALITY
SQuALITY Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Microsoft | 4B | — | — | ||
| 2 | Microsoft | 60B | — | — | ||
| 3 |  Amazon | — | — | — | ||
| 4 | Amazon | — | — | — | ||
| 5 | Amazon | — | — | — |
What is SQuALITY?
SQuALITY (Summarization-format QUestion Answering with Long Input Texts, Yes!) is a long-document summarization dataset built by hiring highly-qualified contractors to read public-domain short stories (3000-6000 words) and write original summaries from scratch. Each document has five summaries: one overview and four question-focused summaries. Designed to address limitations in existing summarization datasets by providing high-quality, faithful summaries.
SQuALITY is a text benchmark evaluating models on language, long context, and summarization tasks. LLM Stats tracks 5 models on this benchmark, scored on a 0–1 scale. The current average is 0.2, with the leader at 0.2.
Compare leaders on the best AI for language, best AI for long context and best AI for summarization leaderboards.
Current leaders
Phi-3.5-mini-instruct from Microsoft currently leads the SQuALITY leaderboard with a score of 0.243 across 5 evaluated AI models.
Source paper
- Title
- SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
- Authors
- Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and 1 others
- Published
- arXiv
- 2205.11465
Abstract
Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries -- which are nearly always in difficult-to-work-with technical domains -- or by using approximate heuristics to extract them from everyday text -- which frequently yields unfaithful summaries. In this work, we turn to a slower but more straightforward approach to developing summarization benchmark data: We hire highly-qualified contractors to read stories and write original summaries from scratch. To amortize reading time, we collect five summaries per document, with the first giving an overview and the subsequent four addressing specific questions. We use this protocol to collect SQuALITY, a dataset of question-focused summaries built on the same public-domain short stories as the multiple-choice dataset QuALITY (Pang et al., 2021). Experiments with state-of-the-art summarization systems show that our dataset is challenging and that existing automatic evaluation metrics are weak indicators of quality.
FAQ
Common questions about the SQuALITY benchmark and leaderboard.