The AI arena is free today

Open Superagent

AlpacaEval 2.0

Paper

Progress Over Time

Interactive timeline showing model performance evolution on AlpacaEval 2.0

State-of-the-art frontier
Open
Proprietary

AlpacaEval 2.0 Leaderboard

4 models
ContextCostLicense
18B——
18B——
3236B——
47B——
Notice missing or incorrect data?
About this benchmark

What is AlpacaEval 2.0?

AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality.

AlpacaEval 2.0 is a text benchmark evaluating models on reasoning, general, creativity, and writing tasks. LLM Stats tracks 4 models on this benchmark, scored on a 0–1 scale. The current average is 0.5, with the leader at 0.6.

Compare leaders on the best AI for reasoning, best AI for general, best AI for creativity and best AI for writing leaderboards.

Current leaders

Granite 3.3 8B Base from IBM currently leads the AlpacaEval 2.0 leaderboard with a score of 0.627 across 4 evaluated AI models.

Source paper

Title
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Authors
Yann Dubois, Balázs Galambosi, Percy Liang, Tatsunori B. Hashimoto
Published
Abstract

LLM-based auto-annotators have become a key component of the LLM development process due to their cost-effectiveness and scalability compared to human-based evaluation. However, these auto-annotators can introduce biases that are hard to remove. Even simple, known confounders such as preference for longer outputs remain in existing automated evaluation metrics. We propose a simple regression analysis approach for controlling biases in auto-evaluations. As a real case study, we focus on reducing the length bias of AlpacaEval, a fast and affordable benchmark for instruction-tuned LLMs that uses LLMs to estimate response quality. Despite being highly correlated with human preferences, AlpacaEval is known to favor models that generate longer outputs. We introduce a length-controlled AlpacaEval that aims to answer the counterfactual question: "What would the preference be if the model's and baseline's output had the same length?" To achieve this, we first fit a generalized linear model to predict the biased auto-annotator's preferences based on the mediators we want to control for (length difference) and other relevant features. We then obtain length-controlled preferences by predicting preferences while conditioning the GLM with a zero difference in lengths. Length-controlling not only improves the robustness of the metric to manipulations in model verbosity, but we also find that it increases the Spearman correlation with LMSYS Chatbot Arena from 0.94 to 0.98.

FAQ

Common questions about the AlpacaEval 2.0 benchmark and leaderboard.

What is the AlpacaEval 2.0 benchmark?

AlpacaEval 2.0 is a length-controlled automatic evaluator for instruction-following language models that uses GPT-4 Turbo to assess model responses against a baseline. It evaluates models on 805 diverse instruction-following tasks including creative writing, classification, programming, and general knowledge questions. The benchmark achieves 0.98 Spearman correlation with ChatBot Arena while being fast (< 3 minutes) and affordable (< $10 in OpenAI credits). It addresses length bias in automatic evaluation through length-controlled win-rates and uses weighted scoring based on response quality.

What is the AlpacaEval 2.0 leaderboard?

The AlpacaEval 2.0 leaderboard ranks 4 AI models based on their performance on this benchmark. Currently, Granite 3.3 8B Base by IBM leads with a score of 0.627. The average score across all models is 0.528.

What is the highest AlpacaEval 2.0 score?

The highest AlpacaEval 2.0 score is 0.627, achieved by Granite 3.3 8B Base from IBM.

How many models are evaluated on AlpacaEval 2.0?

4 models have been evaluated on the AlpacaEval 2.0 benchmark, with 0 verified results and 4 self-reported results.

Where can I find the AlpacaEval 2.0 paper?

The AlpacaEval 2.0 paper is available at https://arxiv.org/abs/2404.04475. The paper details the methodology, dataset construction, and evaluation criteria.

What categories does AlpacaEval 2.0 cover?

AlpacaEval 2.0 is categorized under reasoning, general, creativity, and writing. The benchmark evaluates text models.

What is the best open-source model on AlpacaEval 2.0?

Granite 3.3 8B Base by IBM is the top-ranked open-source model on AlpacaEval 2.0, with a score of 0.627 (rank #1).

How recent are the AlpacaEval 2.0 leaderboard results?

The AlpacaEval 2.0 leaderboard was last updated in November 2026 and currently includes 4 evaluated models.