The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models

TL;DR

BFCL is a benchmark that evaluates LLMs' function calling and tool use abilities, going beyond existing evaluations to measure LLMs' practical capabilities from an agentic perspective.

Problem

While LLMs' ability to actually call tools and execute functions is important, existing benchmarks only evaluate simple function calls or fail to reflect real agent tasks. BFCL addresses the need to comprehensively evaluate LLMs' practical tool use abilities by including diverse function calling scenarios and agent tasks.

Approach

BFCL constructs a dataset that includes multiple function calling types (single function, multiple functions, nested functions, etc.) and agent tasks (tool use, planning, etc.). It measures accuracy, completeness, efficiency, etc. by comparing LLM outputs with ground truth. Additionally, it evaluates various LLMs to analyze differences in function calling abilities.

Results & Contribution

BFCL provides a comprehensive evaluation of LLMs' function calling abilities and clearly shows performance differences among models. This benchmark contributes to research aimed at improving LLMs' practical tool use abilities and provides a reference point for future LLM-based agent development.