Why not average normalized benchmark scores?

Score intervals do not have consistent meaning across benchmarks. The rating approach uses relative performance instead of assuming that one point is universally comparable.

Do missing benchmarks count as zero?

No. Missing evidence increases uncertainty; it is never converted into an invented failure.

Do self-reported scores count?

Yes, with visible provenance. A verified LLM Stats result supersedes the self-reported value for the same model and benchmark.

Why isn't a benchmark I know in the score?

Being listed and being rated are separate decisions. A benchmark feeds the overall score only once it is in the core set, and feeds a category index only after review. Its page states which it is.

How do I get a benchmark listed?

Send the submission items from the Eligibility section. Listing with honest labels is quick; promotion into an index follows the review bar and ships in a versioned index release with a published rank diff.

LLM Stats Score — Methodology, Rating System & Citation

Overview

The LLM Stats Score combines comparable benchmark evidence into one conservative, uncertainty-aware view of model performance.

Missing results are not scored as failures. Models with less evidence retain greater uncertainty, keeping incomplete coverage distinct from observed performance.

Score construction

Benchmark scales do not carry the same meaning, so we compare relative performance within eligible benchmarks before combining evidence across the catalog.

Ties and metric direction are handled consistently while original benchmark values remain visible. The score favors broad, strong evidence and remains cautious when coverage is limited.

Evidence policy

Eligibility and missingness

A benchmark is rated only after review. Each one carries an index scope: listed only, its category indexes, or the core set that feeds the overall score. Being in the catalog does not put a benchmark in the score. Missing results remain missing and are never turned into synthetic losses.

Provenance and precedence

Lab-reported values remain labeled and linked to their source. Results produced by a third-party evaluator are labeled with that evaluator. When LLM Stats independently verifies a result, the verified measurement supersedes both.

Eligibility

New benchmarks do not change the leaderboard on arrival. Each benchmark is reviewed into one of three tiers; its page shows which tier it holds.

Listed only

Benchmark page and scores on model pages, labeled by source. Feeds no index. Every new submission starts here.

Category-scored

Counts toward the category indexes it is tagged with, not the overall score. Requires the review bar below.

Core

Counts toward the overall LLM Stats Score and its categories. Reserved for broadly relevant, widely reported benchmarks, changed only in a versioned index release.

Review bar for a rated tier

At least 10 models from at least 4 organizations, all run on the same harness with prompt, tools, settings and run dates documented.
A frozen version with a task count. A changed task set is a new benchmark.
Uncertainty reported through repeated runs or confidence intervals, or at least 100 items.
Reproducible by LLM Stats, or a sample re-run by us lands within tolerance.
No unresolved conflict of interest. A submitter's own system is never imported as a third-party row, and vendor-authored benchmarks say so.
Not a near-duplicate of a benchmark already rated.

The core set additionally requires broad relevance and coverage of at least 15 models from 5 organizations.

Submitting a benchmark

Name, version, date frozen, task count, and whether tasks are public.
Grading method: deterministic, human, or LLM judge and how disagreements resolve.
Harness: prompt, tools, reasoning settings, runs per model, run dates.
Exact API model identifiers with per-model and per-task results.
Who ran it, who paid for it, and any commercial interest in the outcome.

Promotions and removals ship together in an index release: the methodology version is bumped, the changelog below gains an entry, and the rank changes are published with it.

Category indexes

Category indexes apply the same scoring principles to relevant benchmark subsets. They accept category-scored benchmarks as well as the core set, so a domain benchmark can move its own index without touching the overall score.

Reasoning, Coding, Agent, Math, Long Context, and Vision provide independent views. A model can be highly rated in one category and remain unrated in another; absence means insufficient evidence, not a score of zero.

Limitations

Coverage

The score reflects the evidence available today. It cannot represent capabilities absent from the evaluation set and is not a task-specific deployment recommendation.

Correlated evidence

Related benchmarks can emphasize areas with denser public evaluation. We review the catalog as coverage evolves.

Reporting effects

Self-reported scores depend on evaluation choices. Provenance is visible, but labels cannot remove every source of variation.

Interpretation

Displayed uncertainty describes evidence inside this rating system, not every real-world task or future model behavior.

Changelog

Changes to which benchmarks feed the score, newest first. Earlier versions predate this changelog.

v3.1
November 3, 2026
Benchmark admission tiers
Every benchmark now carries an explicit scope: listed only, category indexes, or the core set that feeds the overall score. New submissions start as listed only and are promoted through review. Third-party evaluations are labeled by evaluator instead of appearing as self-reported. Community hub leaderboards that are not linked to a catalog benchmark count toward category indexes only.
Ranking impact: None for catalog benchmarks: every benchmark already in the score stayed in the core set. Unlinked community leaderboards left the overall score.

Further details

Need more information?

We share additional methodology details with researchers, customers, and partners when appropriate. Contact us to discuss evaluation design, score interpretation, provenance, or a specific use case.

Submit a benchmark or contact us

Interpretation notes

We compare relative benchmark performance because score intervals do not carry consistent meaning across different evaluations.

Missing benchmarks do not count as losses. Lack of evaluation should reduce confidence, not be confused with observed failure.