- Organizations
- XLANG Lab
XLANG Lab: AI benchmarks & research
University of Hong Kong NLP lab. Maintains OSWorld and its variants.
Maintained by XLANG Lab
8 benchmarks- OSWorld
OSWorld: The first-of-its-kind scalable, real computer environment for multimodal agents, supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS with 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows
- Leader
Seed 2.1 Pro78.8%- Models scored
- 21
- OSWorld 2.0
OSWorld 2.0 is a benchmark of 108 long-horizon, real-world computer-use workflows spanning everyday and professional tasks. Each task is an end-to-end workflow that takes human users a median of about 1.6 hours, scored with a binary-completion metric, and targets challenges such as dynamic environments, cross-source reasoning, and implicit-state inference.
- Leader
Claude Opus 5.581.8%
- Models scored
- 17
- OSWorld 2.1 (partial)
Partial-credit computer-use score on OSWorld 2.1, as reported in the Claude Sonnet 5.5 release. Separate from OSWorld 2.0 and strict completion scores.
- Leader
Claude Fable 5.289.3%
- Models scored
- 11
- OSWorld 2.1 (strict)
Strict pass rate on OSWorld 2.1: fraction of tasks where every graded checkpoint is satisfied. Separate from the OSWorld 2.1 partial-credit score and from OSWorld 2.0.
- Leader
Claude Fable 5.254.6%
- Models scored
- 8
- OSWorld Extended
OSWorld is a scalable, real computer environment benchmark for evaluating multimodal agents on open-ended tasks across Ubuntu, Windows, and macOS. It comprises 369 computer tasks involving real web and desktop applications, OS file I/O, and multi-application workflows. The benchmark evaluates agents' ability to interact with computer interfaces using screenshots and actions in realistic computing environments.
- Leader
Claude 3.5 Sonnet22.0%
- Models scored
- 1
- OSWorld-G
OSWorld-G (Grounding) evaluates screenshot grounding accuracy for OS automation tasks.
- Leader
Qwen3 VL 235B A22B Thinking0.7%- Models scored
- 1
- OSWorld Screenshot-only
OSWorld Screenshot-only: A variant of the OSWorld benchmark that evaluates multimodal AI agents using only screenshot observations to complete open-ended computer tasks across real operating systems (Ubuntu, Windows, macOS). Tests agents' ability to perform complex workflows involving web apps, desktop applications, file I/O, and multi-application tasks through visual interface understanding and GUI grounding.
- Leader
Claude 3.5 Sonnet14.9%
- Models scored
- 1
- OSWorld-Verified
OSWorld-Verified is a verified subset of OSWorld, a scalable real computer environment for multimodal agents supporting task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS.
- Leader
Qwen3.8 Max86.1%- Models scored
- 28