The AI arena is free today

Open Superagent

Ramp SWE-Bench v2

Paper

Progress Over Time

Interactive timeline showing model performance evolution on Ramp SWE-Bench v2

State-of-the-art frontier
Open
Proprietary

Ramp SWE-Bench v2 Leaderboard

16 models
ContextCostLicense
1
OpenAI
OpenAI
—1.1M$8.00 / $40.00
2—1.0M$2.00 / $10.00
3
OpenAI
OpenAI
—1.1M$2.00 / $10.00
3—1.0M$8.00 / $40.00
5—1.0M$1.50 / $7.50
6—1.1M$1.00 / $5.00
7—1.0M$4.00 / $20.00
8—500K$2.00 / $6.00
9—1.1M$10.00 / $50.00
10—1.0M$2.00 / $10.00
11763B1.0M$0.30 / $1.20
12—1.1M$2.00 / $10.00
13—1.0M$4.00 / $20.00
14—1.0M$0.10 / $0.50
15—1.1M$0.10 / $0.50
16—1.0M$0.10 / $0.50
Notice missing or incorrect data?

Sub-benchmarks

About this benchmark

What is Ramp SWE-Bench v2?

Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with each model running in its lab's native agent harness (Ramp's v2.1 Native view).

Ramp SWE-Bench v2 is a text benchmark evaluating models on agents and code tasks. LLM Stats tracks 16 models on this benchmark, scored on a 0–1 scale. The current average is 0.7, with the leader at 0.8.

Compare leaders on the best AI for agents and best AI for code leaderboards.

Current leaders

GPT-6.2 Astra from OpenAI currently leads the Ramp SWE-Bench v2 leaderboard with a score of 0.764 across 16 evaluated AI models.

1GPT-6.2 AstraOpenAI76.4%
2Gemini 4 CarbonGoogle75.4%
3GPT-6.2 SolOpenAI73.4%
OSSDeepSeek-V4.1-Flash-1017#11 open-weight60.8%

FAQ

Common questions about the Ramp SWE-Bench v2 benchmark and leaderboard.

What is the Ramp SWE-Bench v2 benchmark?

Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with each model running in its lab's native agent harness (Ramp's v2.1 Native view).

What is the Ramp SWE-Bench v2 leaderboard?

The Ramp SWE-Bench v2 leaderboard ranks 16 AI models based on their performance on this benchmark. Currently, GPT-6.2 Astra by OpenAI leads with a score of 0.764. The average score across all models is 0.650.

What is the highest Ramp SWE-Bench v2 score?

The highest Ramp SWE-Bench v2 score is 0.764, achieved by GPT-6.2 Astra from OpenAI.

How many models are evaluated on Ramp SWE-Bench v2?

16 models have been evaluated on the Ramp SWE-Bench v2 benchmark, with 0 verified results and 0 self-reported results.

Where can I find the Ramp SWE-Bench v2 paper?

The Ramp SWE-Bench v2 paper is available at https://labs.ramp.com/swebench. The paper details the methodology, dataset construction, and evaluation criteria.

What categories does Ramp SWE-Bench v2 cover?

Ramp SWE-Bench v2 is categorized under agents and code. The benchmark evaluates text models.

Are there variants of Ramp SWE-Bench v2?

Yes. Ramp SWE-Bench v2 has 1 related variant: Ramp SWE-Bench v2 (Bash).

What is the best open-source model on Ramp SWE-Bench v2?

DeepSeek-V4.1-Flash-1017 by DeepSeek is the top-ranked open-source model on Ramp SWE-Bench v2, with a score of 0.608 (rank #11).

Which model offers the best value on Ramp SWE-Bench v2?

Among models scoring within 10% of the leader, GPT-6.1 Terra from OpenAI is the cheapest, at $1.00 per million input tokens with a score of 0.704.

How recent are the Ramp SWE-Bench v2 leaderboard results?

The Ramp SWE-Bench v2 leaderboard was last updated in November 2026 and currently includes 16 evaluated models.