SuperGLUE
What is SuperGLUE?
SuperGLUE is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard. It includes 8 primary tasks: BoolQ (Boolean Questions), CB (CommitmentBank), COPA (Choice of Plausible Alternatives), MultiRC (Multi-Sentence Reading Comprehension), ReCoRD (Reading Comprehension with Commonsense Reasoning), RTE (Recognizing Textual Entailment), WiC (Word-in-Context), and WSC (Winograd Schema Challenge). The benchmark evaluates diverse language understanding capabilities including reading comprehension, commonsense reasoning, causal reasoning, coreference resolution, textual entailment, and word sense disambiguation across multiple domains.
SuperGLUE is a text benchmark evaluating models on reasoning, language, and general tasks. LLM Stats tracks 1 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 0.8.
Compare leaders on the best AI for reasoning, best AI for language and best AI for general leaderboards.
Current leaders
o1-mini from OpenAI currently leads the SuperGLUE leaderboard with a score of 0.750 across 1 evaluated AI models.
Source paper
- Title
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- Authors
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, and 4 others
- Published
- arXiv
- 1905.00537
Abstract
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.
FAQ
Common questions about the SuperGLUE benchmark and leaderboard.