SQuALITY
SQuALITY (Summarization-format QUestion Answering with Long Input Texts, Yes!) is a long-document summarization dataset built by hiring highly-qualified contractors to read public-domain short stories (3000-6000 words) and write original summaries from scratch. Each document has five summaries: one overview and four question-focused summaries. Designed to address limitations in existing summarization datasets by providing high-quality, faithful summaries.
Phi-3.5-mini-instruct from Microsoft currently leads the SQuALITY leaderboard with a score of 0.243 across 5 evaluated AI models.
What SQuALITY measures
SQuALITY is a text benchmark that evaluates large language models on language, long context, and summarization tasks. LLM Stats tracks 5 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.2, with the leader reaching 0.2.
Compare leaders on the best AI for language, best AI for long context and best AI for summarization leaderboards.
Publication
- Paper
- SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
- Authors
- Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and 1 others
- Published
- arXiv
- 2205.11465
Abstract
Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries -- which are nearly always in difficult-to-work-with technical domains -- or by using approximate heuristics to extract them from everyday text -- which frequently yields unfaithful summaries. In this work, we turn to a slower but more straightforward approach to developing summarization benchmark data: We hire highly-qualified contractors to read stories and write original summaries from scratch. To amortize reading time, we collect five summaries per document, with the first giving an overview and the subsequent four addressing specific questions. We use this protocol to collect SQuALITY, a dataset of question-focused summaries built on the same public-domain short stories as the multiple-choice dataset QuALITY (Pang et al., 2021). Experiments with state-of-the-art summarization systems show that our dataset is challenging and that existing automatic evaluation metrics are weak indicators of quality.
Phi-3.5-mini-instruct leads with 24.3%, followed by Phi-3.5-MoE-instruct at 24.1% and 
Nova Pro at 19.8%.
Progress Over Time
Interactive timeline showing model performance evolution on SQuALITY
SQuALITY Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | Microsoft | 4B | — | — | ||
| 2 | Microsoft | 60B | — | — | ||
| 3 | Amazon | — | — | — | ||
| 4 | Amazon | — | — | — | ||
| 5 | Amazon | — | — | — |
FAQ
Common questions about SQuALITY.
More evaluations to explore
Related benchmarks in the same category
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.