“They basically created their own API for the sake of this testing, and then did this like mapping to create a graph edge construction and like generate tasks through that.”
quote is from the automated transcript, cleaned for reading:
filler sounds and stutters are removed, nothing is rephrased. names can be misheard
(the analysis reads context, assessments check outside sources). how →
More from Sam Julien
Opinion
BFCL v1 introduced AST-based evaluation for executable functions
“And then the main, like, I think sort of the main innovation of this first version of the leaderboard was that it used this, it introduced this like abstract syntax tree. Evaluation where it would look at the executable functions and evaluate them”
Sam JulienOct 5, 2024▶ 3:52[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer
AssertionContradicted
Julien: BFCL v1 was the first dedicated LLM function calling benchmark
“So the first one that came out, like I said, in March was really the first of its kind to Make a function calling leaderboard.”
Sam JulienOct 5, 2024▶ 2:10[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer
AssertionSupported
Julien: BFCL v3 shifted from AST to state-based evaluation
“So they really updated their evaluation from the abstract syntax tree to something called state-based evaluation, which we'll, we'll talk about in just a second.”
Sam JulienOct 5, 2024▶ 8:24[Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer
Made with StarZero
Turn any episode into a week of clips.
This entire site, over 200 episodes transcribed, diarized, checked and made playable,
runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the
moments worth sharing, cuts them, captions them, and reframes them for every feed.
We use essential cookies to make the site work. With your permission we
also use analytics cookies (Google Analytics and Mixpanel) to understand
usage and improve StarZero. See our Cookie Policy.
