B3 AI Security Benchmark
Progress Over Time
Interactive timeline showing model performance evolution on B3 AI Security Benchmark
B3 AI Security Benchmark Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 | Mistral AI | 1.1T | 1.0M | $0.70 / $2.10 |
What is B3 AI Security Benchmark?
B3 AI Security Benchmark evaluates agent-backbone robustness against contextual prompt-injection attacks collected in Lakera's Agent Breaker Challenge.
B3 AI Security Benchmark is a text benchmark evaluating models on safety and agents tasks. LLM Stats tracks 1 models on this benchmark, scored on a 0–1 scale. The current average is 1.0, with the leader at 1.0.
Compare leaders on the best AI for safety and best AI for agents leaderboards.
Current leaders
Mistral Large 4 from Mistral AI currently leads the B3 AI Security Benchmark leaderboard with a score of 0.971 across 1 evaluated AI models.
Source paper
- Title
- Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
- Authors
- Julia Bazinska, Max Mathys, Francesco Casucci, Mateo Rojas-Carulla, and 3 others
- Published
- arXiv
- 2510.22620
Abstract
AI agents powered by large language models (LLMs) are being deployed at scale, yet we lack a systematic understanding of how the choice of backbone LLM affects agent security. The non-deterministic sequential nature of AI agents complicates security modeling, while the integration of traditional software with AI components entangles novel LLM vulnerabilities with conventional security risks. Existing frameworks only partially address these challenges as they either capture specific vulnerabilities only or require modeling of complete agents. To address these limitations, we introduce threat snapshots: a framework that isolates specific states in an agent's execution flow where LLM vulnerabilities manifest, enabling the systematic identification and categorization of security risks that propagate from the LLM to the agent level. We apply this framework to construct the $b^3$ benchmark, a security benchmark based on 194,331 unique crowdsourced adversarial attacks. We then evaluate 34 popular LLMs with it, revealing, among other insights, that enhanced reasoning capabilities improve security, while model size does not correlate with security. We release our benchmark, dataset, and evaluation code to facilitate widespread adoption by LLM providers and practitioners, offering guidance for agent developers and incentivizing model developers to prioritize backbone security improvements.
FAQ
Common questions about the B3 AI Security Benchmark benchmark and leaderboard.