Translation en→Set1 spBleu
Translation evaluation using spBLEU (SentencePiece BLEU), a BLEU metric computed over text tokenized with a language-agnostic SentencePiece subword model. Introduced in the FLORES-101 evaluation benchmark for low-resource and multilingual machine translation.
Nova Pro from Amazon currently leads the Translation en→Set1 spBleu leaderboard with a score of 0.434 across 3 evaluated AI models.
What Translation en→Set1 spBleu measures
Translation en→Set1 spBleu is a text benchmark that evaluates large language models on language tasks. LLM Stats tracks 3 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.4, with the leader reaching 0.4.
Compare leaders on the best AI for language leaderboards.
Publication
- Paper
- The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
- Authors
- Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, and 6 others
- Published
- arXiv
- 2106.03193
Abstract
One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the FLORES-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are multilingually aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.
Nova Pro leads with 43.4%, followed by Nova Lite at 41.5% and Nova Micro at 40.2%.
Progress Over Time
Interactive timeline showing model performance evolution on Translation en→Set1 spBleu
Translation en→Set1 spBleu Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | Amazon | — | — | — | ||
| 2 | Amazon | — | — | — | ||
| 3 | Amazon | — | — | — |
FAQ
Common questions about Translation en→Set1 spBleu.
More evaluations to explore
Related benchmarks in the same category
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.
WinoGrande: An Adversarial Winograd Schema Challenge at Scale. A large-scale dataset of 44,000 pronoun resolution problems designed to test machine commonsense reasoning. Uses adversarial filtering to reduce spurious biases and provides a more robust evaluation of whether AI systems truly understand commonsense or exploit statistical shortcuts. Current best AI methods achieve 59.4-79.1% accuracy, significantly below human performance of 94.0%.