Assertion Contradicted AI assessment confidence: 95% certainty 4/5 debate potential 1/5

Julien: BFCL v1 was the first dedicated LLM function calling benchmark

Sam Julien · [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer · Oct 5, 2024 · at 2:10

Sam Julien, Head of DevRel at Writer, presents the origins and architecture of the Berkeley Function Calling Leaderboard during a Latent Space Paper Club session.

0:00 / 0:12exact quote · 12.2s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“So the first one that came out, like I said, in March was really the first of its kind to Make a function calling leaderboard.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

The Berkeley Function Calling Leaderboard (BFCL v1), released in March 2024, was the first benchmark and leaderboard of its kind dedicated to evaluating LLM function calling.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

The Berkeley Function Calling Leaderboard (BFCL v1) was neither released in March 2024 (it officially launched on February 26, 2024) nor was it the first benchmark and leaderboard dedicated to evaluating LLM function calling. The Nexus Function Calling Leaderboard (NFCL) was released in December 2023 by Nexusflow as an interactive public leaderboard specifically evaluating function calling across single-turn, nested, and parallel calls (a prior benchmark explicitly cited in the BFCL paper itself). Additionally, earlier dedicated tool- and API-calling evaluation suites with leaderboards, such as ToolBench/ToolEval (July/August 2023) and API-Bank (April 2023), predated BFCL by several months.

Nexusflow released the Nexus Function Calling Leaderboard (NFCL) in December 2023 as a dedicated, public interactive evaluation leaderboard for LLM function calling.
The Berkeley Function Calling Leaderboard (BFCL v1) paper explicitly cites the prior existence of the Nexus Function Calling Leaderboard (Srinivasan et al., 2023).
BFCL v1 was officially released on February 26, 2024, not in March 2024 (though cost/latency additions occurred in April 2024).

Checked against

[1] github.com [2] github.com [3] huggingface.co [4] vellum.ai [5] medium.com [6] github.com [7] huggingface.co [8] openreview.net

"Nexus Function Calling" leaderboard date "Berkeley Function Calling Leaderboard" "February 26, 2024" OR "Feb 26, 2024" OR "March 2024" Berkeley Function Calling Leaderboard release date BFCL v1 release date Gorilla Berkeley "Nexus Function Calling Leaderboard" release date OR 2023 "ToolBench" leaderboard "function calling" OR "tool use" 2023 "NexusRaven" "leaderboard" 2023 "API-Bank" leaderboard 2023 "ToolEval" leaderboard 2023

assessed as of 2026-09-02

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

Berkeley Function Calling Leaderboard function calling

⚑ Spot an error? Report it

More from Sam Julien

Opinion

BFCL v1 introduced AST-based evaluation for executable functions

“And then the main, like, I think sort of the main innovation of this first version of the leaderboard was that it used this, it introduced this like abstract syntax tree. Evaluation where it would look at the executable functions and evaluate them”

Sam Julien Oct 5, 2024 ▶ 3:52 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Assertion Supported

Julien: BFCL v3 shifted from AST to state-based evaluation

“So they really updated their evaluation from the abstract syntax tree to something called state-based evaluation, which we'll, we'll talk about in just a second.”

Sam Julien Oct 5, 2024 ▶ 8:24 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Assertion Not yet assessed · timeframe Oct 2024

BFCL v3 generates evaluation tasks using graph edge construction

“They basically created their own API for the sake of this testing, and then did this like mapping to create a graph edge construction and like generate tasks through that.”

Sam Julien Oct 5, 2024 ▶ 10:46 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Julien: BFCL v1 was the first dedicated LLM function calling benchmark

More from Sam Julien

Turn any episode into a week of clips.