The AI arena is free today

Open Superagent
AI OrganizationMaintains 10 benchmarks

Princeton NLP: AI benchmarks & research

Princeton University NLP group. Maintains SWE-bench and its Verified / Multimodal / Multilingual variants, CharXiv and COLLIE.

Maintained by Princeton NLP

10 benchmarks
  • CharXiv-D

    CharXiv-D is the descriptive questions subset of the CharXiv benchmark, designed to assess multimodal large language models' ability to extract basic information from scientific charts. It contains descriptive questions covering information extraction, enumeration, pattern recognition, and counting across 2,323 diverse charts from arXiv papers, all curated and verified by human experts.

    multimodalreasoningstructured outputvisionCounts toward the LLM Stats Score
    Leader
    Seed 2.1 Pro95.5%
    Models scored
    19
  • CharXiv-R

    CharXiv-R is the reasoning component of the CharXiv benchmark, focusing on complex reasoning questions that require synthesizing information across visual chart elements. It evaluates multimodal large language models on their ability to understand and reason about scientific charts from arXiv papers through various reasoning tasks.

    multimodalreasoningvisionCounts toward the LLM Stats Score
    Leader
    Claude Mythos Preview93.2%
    Models scored
    63
  • COLLIE

    COLLIE is a grammar-based framework for systematic construction of constrained text generation tasks. It allows specification of rich, compositional constraints across diverse generation levels and modeling challenges including language understanding, logical reasoning, and semantic planning. The COLLIE-v1 dataset contains 2,080 instances across 13 constraint structures.

    languagereasoningwritingCounts toward the LLM Stats Score
    Leader
    GPT-599.0%
    Models scored
    11
  • SWE-Bench Multimodal

    SWE-Bench Multimodal extends SWE-Bench to evaluate language models on software engineering tasks that involve visual inputs such as screenshots, UI mockups, and diagrams alongside code understanding.

    multimodalreasoningagentscodevisionCounts toward the LLM Stats Score
    Leader
    Claude Opus 5.561.4%
    Models scored
    7
  • SWE-Bench Verified

    With OpenAI

    A verified subset of 500 software engineering problems from real GitHub issues, validated by human annotators for evaluating language models' ability to resolve real-world coding issues by generating patches for Python codebases.

    reasoningfrontend developmentcodeCounts toward the LLM Stats Score
    Leader
    Claude Fable 595.0%
    Models scored
    123
  • SWE-bench Verified (Agentic Coding)

    With OpenAI

    SWE-bench Verified is a human-filtered subset of 500 software engineering problems drawn from real GitHub issues across 12 popular Python repositories. Given a codebase and an issue description, language models are tasked with generating patches that resolve the described problems. This benchmark evaluates AI's real-world agentic coding skills by requiring models to navigate complex codebases, understand software engineering problems, and coordinate changes across multiple functions, classes, and files to fix well-defined issues with clear descriptions.

    reasoningcodeCounts toward the LLM Stats Score
    Leader
    Claude Sonnet 4.577.2%
    Models scored
    2
  • SWE-bench Verified (Agentless)

    With OpenAI

    A human-validated subset of SWE-bench that evaluates language models' ability to resolve real-world GitHub issues using an agentless approach. The benchmark tests models on software engineering problems requiring understanding and coordinating changes across multiple functions, classes, and files simultaneously.

    reasoninggeneralCounts toward the LLM Stats Score
    Leader
    Kimi K2 Instruct51.8%
    Models scored
    2
  • SWE-bench Verified (Multiple Attempts)

    With OpenAI

    SWE-bench Verified is a human-validated subset of 500 test samples from the original SWE-bench dataset that evaluates AI systems' ability to automatically resolve real GitHub issues in Python repositories. Given a codebase and issue description, models must edit the code to successfully resolve the problem, requiring understanding and coordination of changes across multiple functions, classes, and files. The Verified version provides more reliable evaluation through manual validation of test samples.

    reasoningCounts toward the LLM Stats Score
    Leader
    Kimi K2 Instruct71.6%
    Models scored
    1
  • SWE-Dev

    SWE-bench development split consisting of 225 software engineering problems drawn from real GitHub issues across 12 popular Python repositories. Language models are given a codebase along with a description of an issue to be resolved and must edit the codebase to address the issue, often requiring understanding and coordinating changes across multiple functions, classes, and files.

    frontend developmentCounts toward the LLM Stats Score
    Leader
    No scores yet
    Models scored
    0
  • SWE-MM

    SWE-MM evaluates software-engineering agents on repository tasks that require understanding both source code and visual evidence.

    multimodalagentscodevisionCounts toward the LLM Stats Score
    Leader
    Qwen3.8-27B38.6%
    Models scored
    1

In one line

Princeton NLP maintains CharXiv-D (counts toward the LLM Stats Score, 19 models scored), CharXiv-R (counts toward the LLM Stats Score, 63 models scored), COLLIE (counts toward the LLM Stats Score, 11 models scored), SWE-Bench Multimodal (counts toward the LLM Stats Score, 7 models scored), SWE-Bench Verified (counts toward the LLM Stats Score, 123 models scored), SWE-bench Verified (Agentic Coding) (counts toward the LLM Stats Score, 2 models scored), SWE-bench Verified (Agentless) (counts toward the LLM Stats Score, 2 models scored), SWE-bench Verified (Multiple Attempts) (counts toward the LLM Stats Score, 1 model scored), SWE-Dev (counts toward the LLM Stats Score, 0 models scored) and SWE-MM (counts toward the LLM Stats Score, 1 model scored). It ships no models of its own on LLM Stats.

FAQ

Common questions about Princeton NLP.

What is Princeton NLP?

Princeton University NLP group. Maintains SWE-bench and its Verified / Multimodal / Multilingual variants, CharXiv and COLLIE. On LLM Stats it maintains 10 benchmarks, led by SWE-Bench Verified, CharXiv-R and CharXiv-D. It ships no models of its own; each score on its benchmarks carries its own provenance label on the model page.

Which Princeton NLP benchmarks count toward the LLM Stats Score?

SWE-Bench Verified, CharXiv-R, CharXiv-D, COLLIE, SWE-Bench Multimodal, SWE-bench Verified (Agentic Coding) and 4 more count toward the LLM Stats Score. Only benchmarks admitted to the core set move the main leaderboard; the rest are shown for transparency.

Which models lead Princeton NLP's benchmarks?

Claude Fable 5 leads SWE-Bench Verified at 95.0% (123 models scored), Claude Mythos Preview leads CharXiv-R at 93.2% (63 models scored) and Seed 2.1 Pro leads CharXiv-D at 95.5% (19 models scored). Each benchmark page lists the full ranking and how the results were produced.

Where are Princeton NLP's official benchmark results?

Princeton NLP's official site is https://nlp.cs.princeton.edu. Each benchmark page on LLM Stats links to that benchmark's own paper or leaderboard, and shows the scores next to every other benchmark so models can be compared on one page.