Assertion Supported AI assessment confidence: 95% certainty 4/5 debate potential 1/5

Julien: BFCL v3 shifted from AST to state-based evaluation

Sam Julien · [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer · Oct 5, 2024 · at 8:24

Sam Julien presents the architecture of version 3 of the Berkeley Function Calling Leaderboard during a Latent Space Paper Club session.

0:00 / 0:08exact quote · 8.9s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“So they really updated their evaluation from the abstract syntax tree to something called state-based evaluation, which we'll, we'll talk about in just a second.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Berkeley Function Calling Leaderboard v3 updated its evaluation methodology from abstract syntax trees to state-based evaluation.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

Assessment

On September 19, 2024, the Gorilla Team at UC Berkeley officially announced BFCL V3 (Berkeley Function Calling Leaderboard v3). The official release documentation explicitly confirmed the shift from Abstract Syntax Tree (AST) parsing to state-based evaluation, stating: 'Instead of dissecting function parameter pairs using AST and matching them in a list of possible answers, we now verify the actual state of the API system (like file systems or booking systems) after the model runs its functions.'

Gorilla Team at UC Berkeley blog post ('BFCL V3 • Multi-Turn & Multi-Step Function Calling Evaluation', September 19, 2024) documenting the replacement of AST parameter matching with system state verification.
BFCL GitHub repository release notes (September 21, 2024) announcing the new state-based evaluation system for multi-turn and multi-step function calling capabilities.

Checked against

[1] berkeley.edu [2] github.com [3] berkeley.edu [4] github.io [5] readthedocs.io [6] llm-stats.com [7] mlr.press [8] openreview.net

Gorilla Berkeley Function Calling Leaderboard evaluation methodology AST state Berkeley Function Calling Leaderboard v3 state based evaluation Gorilla Berkeley Function Calling Leaderboard v3 blog

assessed as of 2026-09-02

This assessment is an AI reading of a public statement, made against the cited references with the confidence shown. It can be wrong. The quote, the tape and the references are all on this page; if something is off, use report an error below. full disclaimer →

Mentioned

abstract syntax tree Berkeley Function Calling Leaderboard state-based evaluation

⚑ Spot an error? Report it

More from Sam Julien

Opinion

BFCL v1 introduced AST-based evaluation for executable functions

“And then the main, like, I think sort of the main innovation of this first version of the leaderboard was that it used this, it introduced this like abstract syntax tree. Evaluation where it would look at the executable functions and evaluate them”

Sam Julien Oct 5, 2024 ▶ 3:52 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Assertion Contradicted

Julien: BFCL v1 was the first dedicated LLM function calling benchmark

“So the first one that came out, like I said, in March was really the first of its kind to Make a function calling leaderboard.”

Sam Julien Oct 5, 2024 ▶ 2:10 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Assertion Not yet assessed · timeframe Oct 2024

BFCL v3 generates evaluation tasks using graph edge construction

“They basically created their own API for the sake of this testing, and then did this like mapping to create a graph edge construction and like generate tasks through that.”

Sam Julien Oct 5, 2024 ▶ 10:46 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

Julien: BFCL v3 shifted from AST to state-based evaluation

More from Sam Julien

Turn any episode into a week of clips.