Overview
The LLM Stats Score treats every eligible benchmark as a multiplayer competition. TrueSkill combines those rankings into a skill estimate μ and uncertainty σ. The public score is the conservative estimate μ − 3σ.
Missing results are not scored as zero. Instead, models with sparse evidence retain greater uncertainty and receive a larger conservative discount. A benchmark enters the index once at least three models have comparable results, and it contributes exactly one update.
Score construction
Benchmark scales are not comparable. A point on GPQA does not represent the same capability change as a point on SWE-bench. The calculation therefore uses only rank order within each benchmark.
Exact score ties receive the same rank. Lower-is-better metrics are reversed internally while their original values remain visible. Games are ordered by participant count, then stable benchmark identifier.
S = μ − 3σμ is the posterior skill estimate. σ is posterior uncertainty. S is the published conservative rating.
Uncertainty
Every model starts at the same neutral prior: μ = 25 and σ = 25/3. Benchmark outcomes move μ and narrow σ. A model with fewer results can still rank well, but it must overcome a wider uncertainty interval.
Parameters
| Term | Parameter | Value | Purpose |
|---|---|---|---|
| μ₀ | Initial rating | 25 | Neutral starting skill |
| σ₀ | Initial uncertainty | 25 / 3 | Standard TrueSkill prior |
| p | Rating passes | 1 | Each benchmark counted once |
| n | Minimum field | 3 models | Small fields excluded |
| d | Draw probability | 0.05 | Prior probability of a draw |
| S | Published score | μ − 3σ | Conservative skill estimate |
| CI | Confidence interval | μ ± 2σ | 95% posterior interval |
Evidence policy
Eligibility and missingness
Public catalog benchmarks and approved community evaluations are eligible when at least three LLMs have comparable results. A model participates only in benchmarks where it has evidence. Missing results remain missing; they are never converted into zeroes or synthetic losses.
Provenance and precedence
The index uses one canonical value for each model–benchmark pair. Lab-reported values remain labeled and linked to their source. When LLM Stats verifies the same result on its own infrastructure, that measurement takes precedence.
A failed rating game is retried with a high-precision numerical backend. Only successfully processed games count toward the model's coverage.
Category indexes
Category indexes use the same calculation over tagged benchmark subsets. The overall score uses all eligible benchmarks and is not an average of category scores.
Reasoning, Coding, Agent, Math, Long Context, and Vision are independent views of the same evidence system. A model can be highly rated in one category and remain unrated in another; absence means insufficient evidence, not a score of zero.
Limitations
Construct validity
The score represents the benchmarks available today. It cannot recover capabilities that are absent from the evaluation set, and a broad score should not be read as a task-specific deployment recommendation.
Correlated evidence
Closely related benchmarks each contribute once. We do not infer hidden benchmark families, so research areas with denser public evaluation may contribute more evidence.
Reporting effects
Self-reported scores depend on harnesses, prompts, and configuration choices. Provenance is visible, but labels do not remove every source of measurement variation.
Posterior interpretation
TrueSkill uncertainty describes evidence inside this rating model. It is not a confidence interval over every possible real-world task or future model behavior.
Reproduce
- 01Select visible catalog scores and approved public LLM runs.
- 02Keep benchmarks with at least three participating models.
- 03Rank by score while preserving ties and metric direction.
- 04Initialize each model at Rating(25, 25/3).
- 05Sort games by field size, then benchmark identifier.
- 06Update each game once and report μ − 3σ.
Version history
Citation
LLM Stats (2026). LLM Stats Score (v3.0).
https://llm-stats.com/methodology/llm-stats-scoreInterpretation notes
We use rank order instead of averaging normalized scores because score intervals do not carry consistent meaning across benchmarks. Rank aggregation gives up information about margin, but avoids introducing a false common scale.
Missing benchmarks do not count as losses. Their effect is expressed through wider posterior uncertainty. This distinction matters for new models: lack of evaluation should reduce confidence, not be confused with observed failure.
Reference: Herbrich, Minka & Graepel (2006), TrueSkill™: A Bayesian Skill Rating System.
Questions or corrections? Contact us.