The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 6 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 1 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Berkeley Function Calling Leaderboard is unsuitable for realistic agent evaluation
“This BFCL if you just open and check it you would be like, okay, this is okay, but you don't want to use it for realistic evaluation because it's like some math. Problem statement is given. Some simple tool is given just like, let's say some exponential tool, …”
Function calling benchmarks like BFCL are largely saturated
“I find that a lot of these evals are mostly saturated, like for BFCL. All the models are near, near the top. Already. And kind of the errors are more, I would say like just differences in default behaviors. I think most of the models on the leaderboard can kin…”
BFCL v1 introduced AST-based evaluation for executable functions
“And then the main, like, I think sort of the main innovation of this first version of the leaderboard was that it used this, it introduced this like abstract syntax tree. Evaluation where it would look at the executable functions and evaluate them”
Julien: BFCL v1 was the first dedicated LLM function calling benchmark
“So the first one that came out, like I said, in March was really the first of its kind to Make a function calling leaderboard.”