Ramp SWE-Bench v2 (Bash)
Progress Over Time
Interactive timeline showing model performance evolution on Ramp SWE-Bench v2 (Bash)
Ramp SWE-Bench v2 (Bash) Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | OpenAI | — | 1.1M | $8.00 / $40.00 | ||
| 2 | Google | — | 1.0M | $2.00 / $10.00 | ||
| 3 | Anthropic | — | 1.0M | $8.00 / $40.00 | ||
| 4 | GPT-6.2 SolNew OpenAI | — | 1.1M | $2.00 / $10.00 | ||
| 5 | Anthropic | — | 1.0M | $4.00 / $20.00 | ||
| 6 | Google | — | 1.0M | $1.50 / $7.50 | ||
| 7 | Anthropic | — | 1.0M | $10.00 / $50.00 | ||
| 8 | OpenAI | — | 1.1M | $1.00 / $5.00 | ||
| 9 | Meta | — | 1.0M | $4.00 / $20.00 | ||
| 10 | Anthropic | — | 1.0M | $2.00 / $10.00 | ||
| 11 | xAI | — | 500K | $2.00 / $6.00 | ||
| 12 | Google | — | 1.0M | $2.00 / $10.00 | ||
| 13 | OpenAI | — | 1.1M | $10.00 / $50.00 | ||
| 14 | Meta | — | 1.0M | $0.10 / $0.50 | ||
| 15 | DeepSeek | 763B | 1.0M | $0.30 / $1.20 | ||
| 16 | Anthropic | — | 1.0M | $0.10 / $0.50 | ||
| 17 | OpenAI | — | 1.1M | $2.00 / $10.00 | ||
| 18 | Google | — | 1.0M | $0.75 / $3.75 | ||
| 19 | Moonshot AI | 2.8T | 1.0M | $2.00 / $10.00 | ||
| 20 | OpenAI | — | 1.1M | $0.10 / $0.50 | ||
| 21 | Reflection AI | 501B | 1.0M | $0.40 / $1.20 | ||
| 22 | 120B | 262K | $0.20 / $1.00 |
What is Ramp SWE-Bench v2 (Bash)?
Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with every model running in the same bash-only mini-swe-agent harness (Ramp's v2.0 Bash view), which isolates the model from harness differences.
Ramp SWE-Bench v2 (Bash) is a text benchmark evaluating models on agents and code tasks. LLM Stats tracks 22 models on this benchmark, scored on a 0–1 scale. The current average is 0.6, with the leader at 0.8.
Compare leaders on the best AI for agents and best AI for code leaderboards.
Current leaders
GPT-6.2 Astra from OpenAI currently leads the Ramp SWE-Bench v2 (Bash) leaderboard with a score of 0.750 across 22 evaluated AI models.
FAQ
Common questions about the Ramp SWE-Bench v2 (Bash) benchmark and leaderboard.