CoVoST2
CoVoST 2 is a large-scale multilingual speech translation corpus derived from Common Voice, covering translations from 21 languages into English and from English into 15 languages. The dataset contains 2,880 hours of speech with 78K speakers for speech translation research.
Nova 2 Omni from Amazon currently leads the CoVoST2 leaderboard with a score of 0.407 across 4 evaluated AI models.
What CoVoST2 measures
CoVoST2 is a audio benchmark that evaluates large language models on language, speech to text, and audio tasks. LLM Stats tracks 4 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.4, with the leader reaching 0.4.
Compare leaders on the best AI for language, best AI for speech to text and best AI for audio leaderboards.
Publication
- Paper
- CoVoST 2 and Massively Multilingual Speech-to-Text Translation
- Authors
- Changhan Wang, Anne Wu, Juan Pino
- Published
- arXiv
- 2007.10310
Abstract
Speech translation has recently become an increasingly popular topic of research, partly due to the development of benchmark datasets. Nevertheless, current datasets cover a limited number of languages. With the aim to foster research in massive multilingual speech translation and speech translation for low resource language pairs, we release CoVoST 2, a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into 15 languages. This represents the largest open dataset available to date from total volume and language coverage perspective. Data sanity checks provide evidence about the quality of the data, which is released under CC0 license. We also provide extensive speech recognition, bilingual and multilingual machine translation and speech translation baselines with open-source implementation.
Nova 2 Omni leads with 40.7%, followed by 
Gemini 2.0 Flash at 39.2% and Gemma 4 12B at 38.5%.
Progress Over Time
Interactive timeline showing model performance evolution on CoVoST2
CoVoST2 Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | Amazon | — | — | — | ||
| 2 | Google | — | — | — | ||
| 3 | Google | 12B | — | — | ||
| 4 | Google | — | — | — |
FAQ
Common questions about CoVoST2.
More evaluations to explore
Related benchmarks in the same category
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.
WinoGrande: An Adversarial Winograd Schema Challenge at Scale. A large-scale dataset of 44,000 pronoun resolution problems designed to test machine commonsense reasoning. Uses adversarial filtering to reduce spurious biases and provides a more robust evaluation of whether AI systems truly understand commonsense or exploit statistical shortcuts. Current best AI methods achieve 59.4-79.1% accuracy, significantly below human performance of 94.0%.