The AI arena is free today

Open Superagent
AI OrganizationMaintains 5 benchmarks

Stanford University: AI benchmarks & research

Stanford research groups behind CyBench, SciCode, CheXpert, DermMCQA and AlpacaEval.

Maintained by Stanford University

5 benchmarks
  • AlpacaEval 2.0

    AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality.

    Models scored
    4
  • CheXpert CXR

    CheXpert is a large dataset of 224,316 chest radiographs from 65,240 patients for automated chest X-ray interpretation. The dataset includes uncertainty labels for 14 medical observations extracted from radiology reports. It serves as a benchmark for developing and evaluating automated chest radiograph interpretation models.

    Leader
    MedGemma 4B IT48.1%
    Models scored
    1
  • CyBench

    CyBench is a suite of Capture-the-Flag (CTF) challenges measuring agentic cyber attack capabilities. It evaluates dual-use cybersecurity knowledge and measures the 'unguided success rate', where agents complete tasks end-to-end without guidance on appropriate subtasks.

    Models scored
    3
  • DermMCQA

    Dermatology multiple choice question assessment benchmark for evaluating medical knowledge and diagnostic reasoning in dermatological conditions and treatments.

    Leader
    MedGemma 4B IT71.8%
    Models scored
    1
  • SciCode

    SciCode is a research coding benchmark curated by scientists that challenges language models to code solutions for scientific problems. It contains 338 subproblems decomposed from 80 challenging main problems across 16 natural science sub-fields including mathematics, physics, chemistry, biology, and materials science. Problems require knowledge recall, reasoning, and code synthesis skills.

    Leader
    Mistral Large 491.8%
    Models scored
    37

In one line

Stanford University maintains AlpacaEval 2.0 (counts toward the LLM Stats Score, 4 models scored), CheXpert CXR (counts toward the LLM Stats Score, 1 model scored), CyBench (counts toward the LLM Stats Score, 3 models scored), DermMCQA (counts toward the LLM Stats Score, 1 model scored) and SciCode (counts toward the LLM Stats Score, 37 models scored). It ships no models of its own on LLM Stats.

FAQ

Common questions about Stanford University.

What is Stanford University?

Stanford research groups behind CyBench, SciCode, CheXpert, DermMCQA and AlpacaEval. On LLM Stats it maintains 5 benchmarks, led by SciCode, AlpacaEval 2.0 and CyBench. It ships no models of its own; each score on its benchmarks carries its own provenance label on the model page.

Which Stanford University benchmarks count toward the LLM Stats Score?

SciCode, AlpacaEval 2.0, CyBench, CheXpert CXR and DermMCQA count toward the LLM Stats Score. Only benchmarks admitted to the core set move the main leaderboard; the rest are shown for transparency.

Which models lead Stanford University's benchmarks?

Mistral Large 4 leads SciCode at 91.8% (37 models scored), Granite 3.3 8B Base leads AlpacaEval 2.0 at 62.7% (4 models scored) and Claude Mythos Preview leads CyBench at 100.0% (3 models scored). Each benchmark page lists the full ranking and how the results were produced.

Where are Stanford University's official benchmark results?

Stanford University's official site is https://www.stanford.edu. Each benchmark page on LLM Stats links to that benchmark's own paper or leaderboard, and shows the scores next to every other benchmark so models can be compared on one page.