Berkeley Function Calling Leaderboard
product on 1 show · 8 statements across 3 episodes · said 24 times in 6 episodes since 2024
Mentions by year, every show
tap a year for its mentions
Latent Space 24
2025 4 mentions in 1 episode
2024 20 mentions in 5 episodes 4 per episode
- [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer
- Building AGI with OpenAI's Structured Outputs API
- [Paper Club] Who Validates the Validators? Aligning LLM-Judges with Humans (w/ Eugene Yan)
- Building AGI in Real Time (OpenAI Dev Day 2024)
- Agents @ Work: Dust.tt — with Stanislas Polu
- every mention in 2024, scene by scene →
every mention on every show, scene by scene, with the transcript →
6 statements about Berkeley Function Calling Leaderboard, every show
Berkeley Function Calling Leaderboard is unsuitable for realistic agent evaluation
“This BFCL if you just open and check it you would be like, okay, this is okay, but you don't want to use it for realistic evaluation because it's like some math. Problem statement is given. Some simple tool is given just like, let's say some exponential tool, …”
BFCL v3 generates evaluation tasks using graph edge construction
“They basically created their own API for the sake of this testing, and then did this like mapping to create a graph edge construction and like generate tasks through that.”
Julien: BFCL v3 shifted from AST to state-based evaluation
“So they really updated their evaluation from the abstract syntax tree to something called state-based evaluation, which we'll, we'll talk about in just a second.”
BFCL v1 introduced AST-based evaluation for executable functions
“And then the main, like, I think sort of the main innovation of this first version of the leaderboard was that it used this, it introduced this like abstract syntax tree. Evaluation where it would look at the executable functions and evaluate them”
Julien: BFCL v1 was the first dedicated LLM function calling benchmark
“So the first one that came out, like I said, in March was really the first of its kind to Make a function calling leaderboard.”
Function calling benchmarks like BFCL are largely saturated
“I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near, near the top. Already. And kind of the errors are more, I would say like just differences in default behaviors. I think most of the models on the leaderboard can kin…”