SWE-Bench Multimodal
Progress Over Time
Interactive timeline showing model performance evolution on SWE-Bench Multimodal
SWE-Bench Multimodal Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  Anthropic | — | — | — | ||
| 2 | Anthropic | — | 1.0M | $5.00 / $25.00 | ||
| 3 | Anthropic | — | 1.0M | $3.00 / $15.00 |
What is SWE-Bench Multimodal?
SWE-Bench Multimodal extends SWE-Bench to evaluate language models on software engineering tasks that involve visual inputs such as screenshots, UI mockups, and diagrams alongside code understanding.
SWE-Bench Multimodal is a multimodal benchmark evaluating models on multimodal, reasoning, agents, code, and vision tasks. LLM Stats tracks 3 models on this benchmark, scored on a 0–1 scale. The current average is 0.4, with the leader at 0.6.
Compare leaders on the best AI for multimodal, best AI for reasoning, best AI for agents, best AI for code and best AI for vision leaderboards.
Current leaders
Claude Mythos Preview from Anthropic currently leads the SWE-Bench Multimodal leaderboard with a score of 0.590 across 3 evaluated AI models.
FAQ
Common questions about the SWE-Bench Multimodal benchmark and leaderboard.