Virology Capabilities Test
Progress Over Time
Interactive timeline showing model performance evolution on Virology Capabilities Test
Virology Capabilities Test Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | xAI | — | 500K | $2.00 / $6.00 | ||
| 2 | — | — | — |
What is Virology Capabilities Test?
Virology Capabilities Test (VCT) is an expert-level multiple-choice benchmark measuring the capability to troubleshoot complex virology laboratory protocols. It evaluates dual-use biological knowledge relevant to bioweapons development.
Virology Capabilities Test is a text benchmark evaluating models on safety and healthcare tasks. LLM Stats tracks 2 models on this benchmark, scored on a 0–1 scale. The current average is 0.6, with the leader at 0.6.
Compare leaders on the best AI for safety and best AI for healthcare leaderboards.
Current leaders
Grok 4.7 from xAI currently leads the Virology Capabilities Test leaderboard with a score of 0.630 across 2 evaluated AI models.
Source paper
- Title
- Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark
- Authors
- Jasper Götting, Pedro Medeiros, Jon G Sanders, Nathaniel Li, and 5 others
- Published
- arXiv
- 2504.16137
Abstract
We present the Virology Capabilities Test (VCT), a large language model (LLM) benchmark that measures the capability to troubleshoot complex virology laboratory protocols. Constructed from the inputs of dozens of PhD-level expert virologists, VCT consists of $322$ multimodal questions covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories. VCT is difficult: expert virologists with access to the internet score an average of $22.1\%$ on questions specifically in their sub-areas of expertise. However, the most performant LLM, OpenAI's o3, reaches $43.8\%$ accuracy, outperforming $94\%$ of expert virologists even within their sub-areas of specialization. The ability to provide expert-level virology troubleshooting is inherently dual-use: it is useful for beneficial research, but it can also be misused. Therefore, the fact that publicly available models outperform virologists on VCT raises pressing governance considerations. We propose that the capability of LLMs to provide expert-level troubleshooting of dual-use virology work should be integrated into existing frameworks for handling dual-use technologies in the life sciences.
FAQ
Common questions about the Virology Capabilities Test benchmark and leaderboard.