Multi-Challenge
Progress Over Time
Interactive timeline showing model performance evolution on Multi-Challenge
Multi-Challenge Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Amazon | — | — | — | ||
| 2 | Amazon | — | 1.0M | $0.30 / $2.50 | ||
| 3 | Amazon | — | — | — | ||
| 4 |  OpenAI | — | — | — | ||
| 5 |  Alibaba Cloud / Qwen Team | 397B | 262K | $0.45 / $3.00 | ||
| 6 |  ByteDance | — | 256K | $0.25 / $2.00 | ||
| 7 | 550B | 262K | $0.50 / $2.20 | |||
| 8 | Alibaba Cloud / Qwen Team | 1.0T | 256K | $1.20 / $6.00 | ||
| 9 |  StepFun | 10B | — | — | ||
| 10 | Alibaba Cloud / Qwen Team | 122B | 262K | $0.29 / $2.40 | ||
| 11 | ByteDance | — | 256K | $0.10 / $0.40 | ||
| 12 | Alibaba Cloud / Qwen Team | 27B | 262K | $0.26 / $2.60 | ||
| 13 | OpenAI | — | — | — | ||
| 14 | Alibaba Cloud / Qwen Team | 35B | 262K | $0.14 / $1.00 | ||
| 15 | 120B | 262K | $0.09 / $0.40 | |||
| 16 | Alibaba Cloud / Qwen Team | 9B | 262K | $0.10 / $0.15 | ||
| 17 |  Moonshot AI | 1.0T | — | — | ||
| 17 | Moonshot AI | 1.0T | — | — | ||
| 19 |  Microsoft | 1.0T | — | — | ||
| 20 | Alibaba Cloud / Qwen Team | 4B | — | — | ||
| 21 |  MiniMax | 456B | — | — | ||
| 21 | MiniMax | 456B | — | — | ||
| 23 | OpenAI | — | — | — | ||
| 24 | OpenAI | — | — | — | ||
| 25 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 26 | OpenAI | — | — | — | ||
| 27 | 32B | 262K | $0.05 / $0.20 | |||
| 28 | OpenAI | — | 1.0M | $2.00 / $8.00 | ||
| 29 | OpenAI | — | 1.0M | $0.40 / $1.60 | ||
| 30 | Alibaba Cloud / Qwen Team | 2B | — | — | ||
| 31 | Alibaba Cloud / Qwen Team | 800M | — | — | ||
| 32 | OpenAI | — | 1.0M | $0.10 / $0.40 |
What is Multi-Challenge?
MultiChallenge is a realistic multi-turn conversation evaluation benchmark that challenges frontier LLMs across four key categories: instruction retention (maintaining instructions throughout conversations), inference memory (recalling and connecting details from previous turns), reliable versioned editing (adapting to evolving instructions during collaborative editing), and self-coherence (avoiding contradictions in responses). The benchmark evaluates models on sustained, contextually complex dialogues across diverse topics including travel planning, technical documentation, and professional communication.
Multi-Challenge is a text benchmark evaluating models on chat, reasoning, and communication tasks. LLM Stats tracks 32 models on this benchmark, scored on a 0–1 scale. The current average is 0.5, with the leader at 0.8.
Compare leaders on the best AI for chat, best AI for reasoning and best AI for communication leaderboards.
Current leaders
Nova 2 Pro from Amazon currently leads the Multi-Challenge leaderboard with a score of 0.777 across 32 evaluated AI models.
Source paper
- Title
- MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
- Authors
- Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, and 6 others
- Published
- arXiv
- 2501.17399
Abstract
We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.
FAQ
Common questions about the Multi-Challenge benchmark and leaderboard.