SWE-Lancer
Progress Over Time
Interactive timeline showing model performance evolution on SWE-Lancer
SWE-Lancer Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  OpenAI | — | — | — | ||
| 2 | OpenAI | — | — | — | ||
| 3 | OpenAI | — | 128K | $2.50 / $10.00 | ||
| 4 | OpenAI | — | — | — |
What is SWE-Lancer?
A benchmark for evaluating large language models on real-world freelance software engineering tasks from Upwork. Contains over 1,400 tasks valued at $1 million USD total, ranging from $50 bug fixes to $32,000 feature implementations. Includes both independent engineering tasks graded via end-to-end tests and managerial tasks assessed against original engineering managers' choices.
SWE-Lancer is a text benchmark evaluating models on reasoning and code tasks. LLM Stats tracks 4 models on this benchmark, scored on a 0–1 scale. The current average is 0.4, with the leader at 0.7.
Compare leaders on the best AI for reasoning and best AI for code leaderboards.
Current leaders
GPT-5.1 Codex from OpenAI currently leads the SWE-Lancer leaderboard with a score of 0.663 across 4 evaluated AI models.
Source paper
- Title
- SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
- Authors
- Samuel Miserendino, Michele Wang, Tejal Patwardhan, Johannes Heidecke
- Published
- arXiv
- 2502.12115
Abstract
We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at \$1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks--ranging from \$50 bug fixes to \$32,000 feature implementations--and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.
FAQ
Common questions about the SWE-Lancer benchmark and leaderboard.