HumanEval
Progress Over Time
Interactive timeline showing model performance evolution on HumanEval
HumanEval Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  OpenBMB | 9B | — | — | ||
| 2 |  Moonshot AI | 1.0T | — | — | ||
| 3 |  Anthropic | — | — | — | ||
| 4 |  OpenAI | — | — | — | ||
| 5 | Moonshot AI | 1.0T | — | — | ||
| 6 |  Alibaba Cloud / Qwen Team | 32B | — | — | ||
| 7 | OpenAI | — | — | — | ||
| 8 |  Sarvam AI | 30B | — | — | ||
| 9 |  Mistral AI | 123B | — | — | ||
| 9 | Anthropic | — | — | — | ||
| 11 | Alibaba Cloud / Qwen Team | 34B | — | — | ||
| 12 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 13 | 8B | — | — | |||
| 13 | 8B | — | — | |||
| 15 |  Google | — | — | — | ||
| 16 |  DeepSeek | 236B | — | — | ||
| 16 | 405B | — | — | |||
| 16 |  Amazon | — | — | — | ||
| 19 | Mistral AI | 24B | — | — | ||
| 19 |  Meituan | 560B | — | — | ||
| 21 | 70B | — | — | |||
| 21 | Alibaba Cloud / Qwen Team | 33B | — | — | ||
| 21 |  xAI | — | — | — | ||
| 21 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 25 | OpenAI | — | — | — | ||
| 25 | Anthropic | — | — | — | ||
| 27 | OpenAI | — | — | — | ||
| 28 | Google | 27B | — | — | ||
| 29 | OpenAI | — | — | — | ||
| 30 | OpenAI | — | 128K | $10.00 / $30.00 | ||
| 31 | Alibaba Cloud / Qwen Team | 73B | — | — | ||
| 32 | Alibaba Cloud / Qwen Team | 72B | — | — | ||
| 33 | xAI | — | — | — | ||
| 34 | Google | 12B | — | — | ||
| 34 | Amazon | — | — | — | ||
| 36 | Anthropic | — | — | — | ||
| 37 | Mistral AI | 24B | — | — | ||
| 37 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 39 | Google | — | — | — | ||
| 40 | Alibaba Cloud / Qwen Team | 15B | — | — | ||
| 41 |  Microsoft | 15B | — | — | ||
| 42 | 7B | — | — | |||
| 43 | Amazon | — | — | — | ||
| 43 | Mistral AI | 22B | — | — | ||
| 45 | 70B | — | — | |||
| 46 | Alibaba Cloud / Qwen Team | 8B | — | — | ||
| 47 | Alibaba Cloud / Qwen Team | 7B | — | — | ||
| 48 | Anthropic | — | — | — | ||
| 49 | Google | 8B | — | — | ||
| 49 | 2B | — | — |
What is HumanEval?
A benchmark that measures functional correctness for synthesizing programs from docstrings, consisting of 164 original programming problems assessing language comprehension, algorithms, and simple mathematics
HumanEval is a text benchmark evaluating models on reasoning and code tasks. LLM Stats tracks 66 models on this benchmark, scored on a 0–1 scale. The current average is 0.8, with the leader at 1.0.
Compare leaders on the best AI for reasoning and best AI for code leaderboards.
Current leaders
MiniCPM-SALA from OpenBMB currently leads the HumanEval leaderboard with a score of 0.951 across 66 evaluated AI models.
Source paper
- Title
- Evaluating Large Language Models Trained on Code
- Authors
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and 54 others
- Published
- arXiv
- 2107.03374
Abstract
We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.
FAQ
Common questions about the HumanEval benchmark and leaderboard.