The AI arena is free today

Open Superagent
AI OrganizationMaintains 3 benchmarks

Huawei: AI benchmarks & research

Huawei Noah's Ark Lab. Maintains ACEBench, TempCompass and Android in the Zoo.

Maintained by Huawei

3 benchmarks
  • ACEBench

    ACEBench is a comprehensive benchmark for evaluating Large Language Models' tool usage capabilities across three primary evaluation types: Normal (basic tool usage scenarios), Special (tool usage with ambiguous or incomplete instructions), and Agent (multi-agent interactions simulating real-world dialogues). The benchmark covers 4,538 APIs across 8 major domains and 68 sub-domains including technology, finance, entertainment, society, health, culture, and environment, supporting both English and Chinese languages.

    reasoningfinancegeneralhealthcaretool callingCounts toward the LLM Stats Score
    Leader
    Kimi K2-Instruct-090576.5%
    Models scored
    2
  • AITZ_EM

    Android-In-The-Zoo (AitZ) benchmark for evaluating autonomous GUI agents on smartphones. Contains 18,643 screen-action pairs with chain-of-action-thought annotations spanning over 70 Android apps. Designed to connect perception (screen layouts and UI elements) with cognition (action decision-making) for natural language-triggered smartphone task completion.

    multimodalreasoningagentsCounts toward the LLM Stats Score
    Leader
    Qwen2.5 VL 72B Instruct83.2%
    Models scored
    3
  • TempCompass

    TempCompass is a comprehensive benchmark for evaluating temporal perception capabilities of Video Large Language Models (Video LLMs). It constructs conflicting videos that share identical static content but differ in specific temporal aspects to prevent models from exploiting single-frame bias. The benchmark evaluates multiple temporal aspects including action, motion, speed, temporal order, and attribute changes across diverse task formats including multi-choice QA, yes/no QA, caption matching, and caption generation.

    multimodalreasoningvisionCounts toward the LLM Stats Score
    Leader
    Seed 1.886.9%
    Models scored
    4

In one line

Huawei maintains ACEBench (counts toward the LLM Stats Score, 2 models scored), AITZ_EM (counts toward the LLM Stats Score, 3 models scored) and TempCompass (counts toward the LLM Stats Score, 4 models scored). It ships no models of its own on LLM Stats.

FAQ

Common questions about Huawei.

What is Huawei?

Huawei Noah's Ark Lab. Maintains ACEBench, TempCompass and Android in the Zoo. On LLM Stats it maintains 3 benchmarks, led by TempCompass, AITZ_EM and ACEBench. It ships no models of its own; each score on its benchmarks carries its own provenance label on the model page.

Which Huawei benchmarks count toward the LLM Stats Score?

TempCompass, AITZ_EM and ACEBench count toward the LLM Stats Score. Only benchmarks admitted to the core set move the main leaderboard; the rest are shown for transparency.

Which models lead Huawei's benchmarks?

Seed 1.8 leads TempCompass at 86.9% (4 models scored), Qwen2.5 VL 72B Instruct leads AITZ_EM at 83.2% (3 models scored) and Kimi K2-Instruct-0905 leads ACEBench at 76.5% (2 models scored). Each benchmark page lists the full ranking and how the results were produced.

Where are Huawei's official benchmark results?

Huawei's official site is https://www.huawei.com. Each benchmark page on LLM Stats links to that benchmark's own paper or leaderboard, and shows the scores next to every other benchmark so models can be compared on one page.