ECLeKTic
Progress Over Time
Interactive timeline showing model performance evolution on ECLeKTic
ECLeKTic Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Google | 8B | — | — | ||
| 2 | Google | 27B | — | — | ||
| 3 | Google | 12B | — | — | ||
| 4 | Google | 4B | — | — | ||
| 5 | Google | 8B | — | — | ||
| 5 | 2B | — | — | |||
| 7 | 2B | — | — | |||
| 8 | Google | 1B | — | — |
What is ECLeKTic?
A multilingual closed-book question answering dataset that evaluates cross-lingual knowledge transfer in large language models across 12 languages, using knowledge-seeking questions based on Wikipedia articles that exist only in one language
ECLeKTic is a text benchmark evaluating models on language and reasoning tasks. LLM Stats tracks 8 models on this benchmark, scored on a 0–1 scale. The current average is 0.1, with the leader at 0.2.
Compare leaders on the best AI for language and best AI for reasoning leaderboards.
Current leaders
Gemma 3n E4B Instructed from Google currently leads the ECLeKTic leaderboard with a score of 0.190 across 8 evaluated AI models.
Source paper
- Title
- ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer
- Authors
- Omer Goldman, Uri Shaham, Dan Malkin, Sivan Eiger, and 10 others
- Published
- arXiv
- 2502.21228
Abstract
To achieve equitable performance across languages, large language models (LLMs) must be able to abstract knowledge beyond the language in which it was learnt. However, the current literature lacks reliable ways to measure LLMs' capability of such cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. Concretely, we used the presence and absence of Wikipedia articles in 12 languages to detect pieces of information that were likely available during pre-training in one of the languages but not in the others. We curate ECLeKTic as a set of fact-seeking questions over this kind of information, in all the different languages. Therefore, in order to solve ECLeKTic the model is required to transfer knowledge between languages. We evaluated 8 LLMs and showed that current SOTA models struggle to effectively share knowledge across languages, even if they can predict the answer for questions in the language in which the knowledge was acquired.
FAQ
Common questions about the ECLeKTic benchmark and leaderboard.