Ramp SWE-Bench v2
Progress Over Time
Interactive timeline showing model performance evolution on Ramp SWE-Bench v2
Ramp SWE-Bench v2 Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | OpenAI | — | 1.1M | $8.00 / $40.00 | ||
| 2 | Google | — | 1.0M | $2.00 / $10.00 | ||
| 3 | GPT-6.2 SolNew OpenAI | — | 1.1M | $2.00 / $10.00 | ||
| 3 | Anthropic | — | 1.0M | $8.00 / $40.00 | ||
| 5 | Google | — | 1.0M | $1.50 / $7.50 | ||
| 6 | OpenAI | — | 1.1M | $1.00 / $5.00 | ||
| 7 | Anthropic | — | 1.0M | $4.00 / $20.00 | ||
| 8 | xAI | — | 500K | $2.00 / $6.00 | ||
| 9 | OpenAI | — | 1.1M | $10.00 / $50.00 | ||
| 10 | Anthropic | — | 1.0M | $2.00 / $10.00 | ||
| 11 | DeepSeek | 763B | 1.0M | $0.30 / $1.20 | ||
| 12 | OpenAI | — | 1.1M | $2.00 / $10.00 | ||
| 13 | Meta | — | 1.0M | $4.00 / $20.00 | ||
| 14 | Anthropic | — | 1.0M | $0.10 / $0.50 | ||
| 15 | OpenAI | — | 1.1M | $0.10 / $0.50 | ||
| 16 | Meta | — | 1.0M | $0.10 / $0.50 |
Sub-benchmarks
What is Ramp SWE-Bench v2?
Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with each model running in its lab's native agent harness (Ramp's v2.1 Native view).
Ramp SWE-Bench v2 is a text benchmark evaluating models on agents and code tasks. LLM Stats tracks 16 models on this benchmark, scored on a 0–1 scale. The current average is 0.7, with the leader at 0.8.
Compare leaders on the best AI for agents and best AI for code leaderboards.
Current leaders
GPT-6.2 Astra from OpenAI currently leads the Ramp SWE-Bench v2 leaderboard with a score of 0.764 across 16 evaluated AI models.
FAQ
Common questions about the Ramp SWE-Bench v2 benchmark and leaderboard.