LiveCodeBench
Progress Over Time
Interactive timeline showing model performance evolution on LiveCodeBench
LiveCodeBench Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  DeepSeek | 1.6T | 1.0M | $1.30 / $2.60 | ||
| 2 | DeepSeek | 284B | 1.0M | $0.09 / $0.18 | ||
| 3 | DeepSeek | 284B | 1.0M | $0.09 / $0.18 | ||
| 4 |  Upstage | — | 524K | $0.30 / $1.20 | ||
| 5 | DeepSeek | 685B | — | — | ||
| 5 | DeepSeek | 685B | 164K | $0.26 / $0.38 | ||
| 7 |  MiniMax | 230B | 1.0M | $0.30 / $1.20 | ||
| 8 |  Meituan | 560B | — | — | ||
| 9 | 120B | 262K | $0.09 / $0.40 | |||
| 10 |  xAI | — | — | — | ||
| 11 | xAI | — | — | — | ||
| 12 | xAI | — | 128K | $3.00 / $15.00 | ||
| 12 | xAI | — | — | — | ||
| 12 | Meituan | 560B | — | — | ||
| 15 | xAI | — | — | — | ||
| 16 | MiniMax | 230B | 1.0M | $0.30 / $1.20 | ||
| 17 |  Amazon | — | — | — | ||
| 18 | DeepSeek | 685B | — | — | ||
| 19 | DeepSeek | 671B | 164K | $0.50 / $2.15 | ||
| 20 |  Zhipu AI | 355B | — | — | ||
| 21 |  NVIDIA | 9B | — | — | ||
| 22 | Amazon | — | 1.0M | $0.30 / $2.50 | ||
| 23 | Zhipu AI | 106B | — | — | ||
| 23 |  Alibaba Cloud / Qwen Team | 235B | — | — | ||
| 25 | — | — | — | |||
| 26 |  Inception | — | 128K | $0.25 / $0.75 | ||
| 27 | 253B | — | — | |||
| 28 | Alibaba Cloud / Qwen Team | 33B | 128K | $0.08 / $0.28 | ||
| 29 | MiniMax | 456B | — | — | ||
| 30 |  Mistral AI | 14B | — | — | ||
| 31 | Mistral AI | 119B | 256K | $0.15 / $0.60 | ||
| 32 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 33 | Alibaba Cloud / Qwen Team | 31B | 128K | $0.10 / $0.44 | ||
| 34 | MiniMax | 456B | — | — | ||
| 35 | Mistral AI | 8B | — | — | ||
| 36 | DeepSeek | 71B | — | — | ||
| 37 | DeepSeek | 33B | — | — | ||
| 38 | DeepSeek | 671B | 164K | $0.25 / $0.95 | ||
| 39 | Alibaba Cloud / Qwen Team | 73B | 33K | $0.36 / $0.40 | ||
| 40 | Mistral AI | 3B | — | — | ||
| 41 |  Microsoft | 14B | — | — | ||
| 42 |  Moonshot AI | 1.0T | — | — | ||
| 43 | DeepSeek | 15B | — | — | ||
| 43 | Microsoft | 14B | — | — | ||
| 45 | Mistral AI | 24B | — | — | ||
| 46 | Mistral AI | 24B | — | — | ||
| 47 | DeepSeek | 671B | — | — | ||
| 47 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 49 | DeepSeek | 671B | 164K | $0.24 / $0.90 | ||
| 50 | Meituan | 560B | — | — |
What is LiveCodeBench?
LiveCodeBench is a holistic and contamination-free evaluation benchmark for large language models for code. It continuously collects new problems from programming contests (LeetCode, AtCoder, CodeForces) and evaluates four different scenarios: code generation, self-repair, code execution, and test output prediction. Problems are annotated with release dates to enable evaluation on unseen problems released after a model's training cutoff.
LiveCodeBench is a text benchmark evaluating models on reasoning, general, and code tasks. LLM Stats tracks 75 models on this benchmark, scored on a 0–1 scale. The current average is 0.5, with the leader at 0.9.
Compare leaders on the best AI for reasoning, best AI for general and best AI for code leaderboards.
Current leaders
DeepSeek-V4-Pro-Max from DeepSeek currently leads the LiveCodeBench leaderboard with a score of 0.935 across 75 evaluated AI models.
Source paper
- Title
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Authors
- Naman Jain, King Han, Alex Gu, Wen-Ding Li, and 6 others
- Published
- arXiv
- 2403.07974
Abstract
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model
FAQ
Common questions about the LiveCodeBench benchmark and leaderboard.