WMT24++
WMT24++ is a comprehensive multilingual machine translation benchmark that expands the WMT24 dataset to cover 55 languages and dialects. It includes human-written references and post-edits across four domains (literary, news, social, and speech) to evaluate machine translation systems and large language models across diverse linguistic contexts.
Nemotron 3 Super (120B A12B) from NVIDIA currently leads the WMT24++ leaderboard with a score of 0.867 across 20 evaluated AI models.
What WMT24++ measures
WMT24++ is a text benchmark that evaluates large language models on language tasks. LLM Stats tracks 20 models on this benchmark, with a maximum possible score of 1. Current average across reported models is 0.6, with the leader reaching 0.9.
Compare leaders on the best AI for language leaderboards.
Publication
- Paper
- WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects
- Authors
- Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, and 13 others
- Published
- arXiv
- 2502.12404
Abstract
As large language models (LLM) become more and more capable in languages other than English, it is important to collect benchmark datasets in order to evaluate their multilingual performance, including on tasks like machine translation (MT). In this work, we extend the WMT24 dataset to cover 55 languages by collecting new human-written references and post-edits for 46 new languages and dialects in addition to post-edits of the references in 8 out of 9 languages in the original WMT24 dataset. The dataset covers four domains: literary, news, social, and speech. We benchmark a variety of MT providers and LLMs on the collected dataset using automatic metrics and find that LLMs are the best-performing MT systems in all 55 languages. These results should be confirmed using a human-based evaluation, which we leave for future work.
Nemotron 3 Super (120B A12B) leads with 86.7%, followed by Nemotron 3 Nano (30B A3B) at 86.2% and 
Qwen3.7 Max at 85.8%.
Progress Over Time
Interactive timeline showing model performance evolution on WMT24++
WMT24++ Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | 120B | — | — | |||
| 2 | 32B | 262K | $0.06 / $0.24 | |||
| 3 |  Alibaba Cloud / Qwen Team | — | 1.0M | $1.25 / $3.75 | ||
| 4 | Alibaba Cloud / Qwen Team | — | 1.0M | $0.50 / $3.00 | ||
| 5 | Alibaba Cloud / Qwen Team | 397B | 262K | $0.60 / $3.60 | ||
| 6 | Alibaba Cloud / Qwen Team | 122B | 262K | $0.40 / $3.20 | ||
| 7 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.30 / $2.40 | ||
| 8 | Alibaba Cloud / Qwen Team | 35B | 262K | $0.25 / $2.00 | ||
| 9 | Alibaba Cloud / Qwen Team | 9B | — | — | ||
| 10 | Alibaba Cloud / Qwen Team | 4B | — | — | ||
| 11 |  Google | 27B | — | — | ||
| 12 | Google | 12B | — | — | ||
| 13 | 2B | — | — | |||
| 13 | Google | 8B | — | — | ||
| 15 | Google | 4B | — | — | ||
| 16 | Alibaba Cloud / Qwen Team | 2B | — | — | ||
| 17 | Google | 8B | — | — | ||
| 17 | 2B | — | — | |||
| 19 | Google | 1B | — | — | ||
| 20 | Alibaba Cloud / Qwen Team | 800M | — | — |
FAQ
Common questions about WMT24++.
More evaluations to explore
Related benchmarks in the same category
A more robust and challenging multi-task language understanding benchmark that extends MMLU by expanding multiple-choice options from 4 to 10, eliminating trivial questions, and focusing on reasoning-intensive tasks. Features over 12,000 curated questions across 14 domains and causes a 16-33% accuracy drop compared to original MMLU.
Massive Multitask Language Understanding benchmark testing knowledge across 57 diverse subjects including STEM, humanities, social sciences, and professional domains
An improved version of the MMLU benchmark featuring manually re-annotated questions to identify and correct errors in the original dataset. Provides more reliable evaluation metrics for language models by addressing dataset quality issues found in the original MMLU.
Multilingual Massive Multitask Language Understanding dataset released by OpenAI, featuring professionally translated MMLU test questions across 14 languages including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Portuguese, Swahili, Yoruba, and Chinese. Contains approximately 15,908 multiple-choice questions per language covering 57 subjects.
Extended version of MMLU-Pro providing additional challenging multiple-choice questions for evaluating language models across diverse academic and professional domains. Built on the foundation of the Massive Multitask Language Understanding benchmark framework.
BIG-Bench Hard (BBH) is a subset of 23 challenging BIG-Bench tasks selected because prior language model evaluations did not outperform average human-rater performance. The benchmark contains 6,511 evaluation examples testing various forms of multi-step reasoning including arithmetic, logical reasoning (Boolean expressions, logical deduction), geometric reasoning, temporal reasoning, and language understanding. Tasks require capabilities such as causal judgment, object counting, navigation, pattern recognition, and complex problem solving.