The AI arena is free today

Open Superagent
AI OrganizationMaintains 2 benchmarks

Ramp: AI benchmarks & research

Finance automation company. Ramp Labs maintains Ramp SWE-Bench.

Maintained by Ramp

2 benchmarks
  • Ramp SWE-Bench v2

    Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with each model running in its lab's native agent harness (Ramp's v2.1 Native view).

    Leader
    GPT-6.2 Astra76.4%
    Models scored
    16
  • Ramp SWE-Bench v2 (Bash)

    Ramp SWE-Bench v2 is Ramp Labs' private coding benchmark of 500 backend engineering tasks built from pull requests that Ramp's in-house coding agent merged into production after engineer review, across work shipped by more than 200 Ramp engineers. Each model gets a one-shot prompt in the same sandboxed copy of Ramp's codebase and must produce a review-ready patch. A panel of LLM judges compares the candidate patch with the engineer-reviewed merged solution; a task passes when there is no material difference where the merged solution is better. Scores are pass@1 at high reasoning effort with web tools disabled, with every model running in the same bash-only mini-swe-agent harness (Ramp's v2.0 Bash view), which isolates the model from harness differences.

    Leader
    GPT-6.2 Astra75.0%
    Models scored
    22

In one line

Ramp maintains Ramp SWE-Bench v2 (listed only, 16 models scored) and Ramp SWE-Bench v2 (Bash) (listed only, 22 models scored). It ships no models of its own on LLM Stats.

FAQ

Common questions about Ramp.

What is Ramp?

Finance automation company. Ramp Labs maintains Ramp SWE-Bench. On LLM Stats it maintains 2 benchmarks, led by Ramp SWE-Bench v2 (Bash) and Ramp SWE-Bench v2. It ships no models of its own; each score on its benchmarks carries its own provenance label on the model page.

Which Ramp benchmarks count toward the LLM Stats Score?

Ramp SWE-Bench v2 (Bash) and Ramp SWE-Bench v2 are listed only. Only benchmarks admitted to the core set move the main leaderboard; the rest are shown for transparency.

Which models lead Ramp's benchmarks?

GPT-6.2 Astra leads Ramp SWE-Bench v2 (Bash) at 75.0% (22 models scored) and GPT-6.2 Astra leads Ramp SWE-Bench v2 at 76.4% (16 models scored). Each benchmark page lists the full ranking and how the results were produced.

Where are Ramp's official benchmark results?

Ramp's official site is https://ramp.com. Each benchmark page on LLM Stats links to that benchmark's own paper or leaderboard, and shows the scores next to every other benchmark so models can be compared on one page.