The AI arena is free today

Open Superagent

Ramp SWE-Bench v2 (Bash)

Paper

Progress Over Time

Interactive timeline showing model performance evolution on Ramp SWE-Bench v2 (Bash)

State-of-the-art frontier
Open
Proprietary

Ramp SWE-Bench v2 (Bash) Leaderboard

22 models
ContextCostLicense
1
OpenAI
OpenAI
—1.1M$8.00 / $40.00
2—1.0M$2.00 / $10.00
3—1.0M$8.00 / $40.00
4
OpenAI
OpenAI
—1.1M$2.00 / $10.00
5—1.0M$4.00 / $20.00
6—1.0M$1.50 / $7.50
7—1.0M$10.00 / $50.00
8—1.1M$1.00 / $5.00
9—1.0M$4.00 / $20.00
10—1.0M$2.00 / $10.00
11—500K$2.00 / $6.00
12—1.0M$2.00 / $10.00
13—1.1M$10.00 / $50.00
14—1.0M$0.10 / $0.50
15763B1.0M$0.30 / $1.20
16—1.0M$0.10 / $0.50
17—1.1M$2.00 / $10.00
18—1.0M$0.75 / $3.75
19
Moonshot AI
Moonshot AI
2.8T1.0M$2.00 / $10.00
20—1.1M$0.10 / $0.50
21
Reflection AI
Reflection AI
501B1.0M$0.40 / $1.20
22120B262K$0.20 / $1.00
Notice missing or incorrect data?
About this benchmark

What is Ramp SWE-Bench v2 (Bash)?

Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with every model running in the same bash-only mini-swe-agent harness (Ramp's v2.0 Bash view), which isolates the model from harness differences.

Ramp SWE-Bench v2 (Bash) is a text benchmark evaluating models on agents and code tasks. LLM Stats tracks 22 models on this benchmark, scored on a 0–1 scale. The current average is 0.6, with the leader at 0.8.

Compare leaders on the best AI for agents and best AI for code leaderboards.

Current leaders

GPT-6.2 Astra from OpenAI currently leads the Ramp SWE-Bench v2 (Bash) leaderboard with a score of 0.750 across 22 evaluated AI models.

1GPT-6.2 AstraOpenAI75.0%
2Gemini 4 CarbonGoogle74.0%
3Claude Fable 5.2Anthropic73.0%
OSSDeepSeek-V4.1-Flash-1017#15 open-weight53.4%

FAQ

Common questions about the Ramp SWE-Bench v2 (Bash) benchmark and leaderboard.

What is the Ramp SWE-Bench v2 (Bash) benchmark?

Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with every model running in the same bash-only mini-swe-agent harness (Ramp's v2.0 Bash view), which isolates the model from harness differences.

What is the Ramp SWE-Bench v2 (Bash) leaderboard?

The Ramp SWE-Bench v2 (Bash) leaderboard ranks 22 AI models based on their performance on this benchmark. Currently, GPT-6.2 Astra by OpenAI leads with a score of 0.750. The average score across all models is 0.561.

What is the highest Ramp SWE-Bench v2 (Bash) score?

The highest Ramp SWE-Bench v2 (Bash) score is 0.750, achieved by GPT-6.2 Astra from OpenAI.

How many models are evaluated on Ramp SWE-Bench v2 (Bash)?

22 models have been evaluated on the Ramp SWE-Bench v2 (Bash) benchmark, with 0 verified results and 0 self-reported results.

Where can I find the Ramp SWE-Bench v2 (Bash) paper?

The Ramp SWE-Bench v2 (Bash) paper is available at https://labs.ramp.com/swebench. The paper details the methodology, dataset construction, and evaluation criteria.

What categories does Ramp SWE-Bench v2 (Bash) cover?

Ramp SWE-Bench v2 (Bash) is categorized under agents and code. The benchmark evaluates text models.

What's the difference between Ramp SWE-Bench v2 (Bash) and Ramp SWE-Bench v2?

Ramp SWE-Bench v2 (Bash) is a variant of Ramp SWE-Bench v2. See the Ramp SWE-Bench v2 leaderboard for the broader benchmark and per-model comparison.

What is the best open-source model on Ramp SWE-Bench v2 (Bash)?

DeepSeek-V4.1-Flash-1017 by DeepSeek is the top-ranked open-source model on Ramp SWE-Bench v2 (Bash), with a score of 0.534 (rank #15).

Which model offers the best value on Ramp SWE-Bench v2 (Bash)?

Among models scoring within 10% of the leader, Gemini 4 Hydrogen from Google is the cheapest, at $1.50 per million input tokens with a score of 0.708.

How recent are the Ramp SWE-Bench v2 (Bash) leaderboard results?

The Ramp SWE-Bench v2 (Bash) leaderboard was last updated in November 2026 and currently includes 22 evaluated models.