Sam Julien

Everything Sam Julien said on any show that made the record, most notable first. Each card names its show and opens the statement there.

LATENT SPACE Opinion

BFCL v1 introduced AST-based evaluation for executable functions

“And then the main, like, I think sort of the main innovation of this first version of the leaderboard was that it used this, it introduced this like abstract syntax tree. Evaluation where it would look at the executable functions and evaluate them”

Sam Julien Oct 5, 2024 ▶ 3:52 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

LATENT SPACE Assertion Contradicted

Julien: BFCL v1 was the first dedicated LLM function calling benchmark

“So the first one that came out, like I said, in March was really the first of its kind to Make a function calling leaderboard.”

Sam Julien Oct 5, 2024 ▶ 2:10 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

LATENT SPACE Assertion Supported

Julien: BFCL v3 shifted from AST to state-based evaluation

“So they really updated their evaluation from the abstract syntax tree to something called state-based evaluation, which we'll, we'll talk about in just a second.”

Sam Julien Oct 5, 2024 ▶ 8:24 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

LATENT SPACE Assertion Not yet assessed · timeframe Oct 2024

BFCL v3 generates evaluation tasks using graph edge construction

“They basically created their own API for the sake of this testing, and then did this like mapping to create a graph edge construction and like generate tasks through that.”

Sam Julien Oct 5, 2024 ▶ 10:46 [Paper Club] Berkeley Function Calling Paper Club! — Sam Julien, Writer

The other half of the tape: Sam Julien's own voice is left out of every number here. Other people bring the name up 2 times in 1 episode across the shows. every mention, with the transcript →

Every mention by year

One line per show, most statements first. The link opens Sam's full record on that show: the calibration, argument clarity, speaking style and every statement made there.

Show	Role there	Eps	Statements	Record
LATENT SPACELEDGER	Director of Product, CopilotKit	1	4	50% 1/2	full record on Latent Space →