ComplexFuncBench
Progress Over Time
Interactive timeline showing model performance evolution on ComplexFuncBench
ComplexFuncBench Leaderboard
| Context | Cost | License | ||||
|---|---|---|---|---|---|---|
| 1 |  OpenAI | — | 128K | $2.50 / $10.00 | ||
| 2 | OpenAI | — | 1.0M | $2.00 / $8.00 | ||
| 3 |  Amazon | — | 1.0M | $0.33 / $2.75 | ||
| 4 | OpenAI | — | — | — | ||
| 5 | OpenAI | — | 1.0M | $0.40 / $1.60 | ||
| 6 | OpenAI | — | — | — | ||
| 7 | OpenAI | — | 1.0M | $0.10 / $0.40 |
What is ComplexFuncBench?
ComplexFuncBench is a benchmark designed to evaluate large language models' capabilities in handling complex function calling scenarios. It encompasses multi-step and constrained function calling tasks that require long-parameter filling, parameter value reasoning, and managing contexts up to 128k tokens. The benchmark includes 1,000 samples across five real-world scenarios.
ComplexFuncBench is a text benchmark evaluating models on long context, reasoning, structured output, and tool calling tasks. LLM Stats tracks 7 models on this benchmark, scored on a 0–1 scale. The current average is 0.5, with the leader at 0.7.
Compare leaders on the best AI for long context, best AI for reasoning, best AI for structured output and best AI for tool calling leaderboards.
Current leaders
GPT-4o from OpenAI currently leads the ComplexFuncBench leaderboard with a score of 0.665 across 7 evaluated AI models.
Source paper
- Title
- ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario
- Authors
- Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and 1 others
- Published
- arXiv
- 2501.10132
Abstract
Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}.
FAQ
Common questions about the ComplexFuncBench benchmark and leaderboard.