- Organizations
- Stanford University
Stanford University: AI benchmarks & research
Stanford research groups behind CyBench, SciCode, CheXpert, DermMCQA and AlpacaEval.
Maintained by Stanford University
5 benchmarks- AlpacaEval 2.0
AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality.
- Leader
Granite 3.3 8B Base62.7%
- Models scored
- 4
- CheXpert CXR
CheXpert is a large dataset of 224,316 chest radiographs from 65,240 patients for automated chest X-ray interpretation. The dataset includes uncertainty labels for 14 medical observations extracted from radiology reports. It serves as a benchmark for developing and evaluating automated chest radiograph interpretation models.
- Leader
MedGemma 4B IT48.1%
- Models scored
- 1
- CyBench
CyBench is a suite of Capture-the-Flag (CTF) challenges measuring agentic cyber attack capabilities. It evaluates dual-use cybersecurity knowledge and measures the 'unguided success rate', where agents complete tasks end-to-end without guidance on appropriate subtasks.
- Leader
Claude Mythos Preview100.0%
- Models scored
- 3
- DermMCQA
Dermatology multiple choice question assessment benchmark for evaluating medical knowledge and diagnostic reasoning in dermatological conditions and treatments.
- Leader
MedGemma 4B IT71.8%
- Models scored
- 1
- SciCode
SciCode is a research coding benchmark curated by scientists that challenges language models to code solutions for scientific problems. It contains 338 subproblems decomposed from 80 challenging main problems across 16 natural science sub-fields including mathematics, physics, chemistry, biology, and materials science. Problems require knowledge recall, reasoning, and code synthesis skills.
- Leader
Mistral Large 491.8%
- Models scored
- 37