Jul 14, 2025 · 34m · latent-space
⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Lightning Pod, hosts Swyx and Alessio Fanelli interview Pratik Bhavsar of Galileo Labs to explore the methodologies, surprising model rankings, and future multi-turn simulation architectures shaping AI agent evaluation.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 16.6% of the talking time here. How this is scored →
The hosts as informed peer 4.3 Guest teaching 3.9 Guest disagreement 0.3 The hosts pushing back 0.7
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:00 Firm correction on Mistral vs Meta models
When Swyx suggests Pratik misspoke and meant Meta rather than Mistral, Pratik immediately corrects the premise, reaffirming that Mistral 3.1 performed well while Llama models struggled.
Hardest push from the hosts ▶ 9:59 Swyx questions guest's model attributionSwyx interrupts the flow to directly challenge Pratik's statement, asking if he actually meant Meta instead of Mistral.
Biggest teaching moment ▶ 7:59 Explaining reasoning models' failure in multi-tool callingPratik breaks down counterintuitive benchmark data explaining how OpenAI o-series models underperformed because they collapsed multiple required tool executions into single calls.
The host holds their own ▶ 13:14 Swyx introduces the freshly released Tau-squared benchmarkSwyx displays superior immediate awareness of the eval landscape by informing the guest about the previous day's Tau-squared release from Sierra, including its newly added telecom domain.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| The Shift from General LLMs to Agent Evaluations | 5 | 1 | 0 | 0 | Swyx frames the shift in LLM evaluation towards agentic tool calling and emphasizes the importance of cost-awareness metrics. Pratik completely agrees and corroborates with customer trends seen at Galileo. | |
| Leaderboard Architecture and Tool Selection Quality Metric | 3 | 3 | 0 | 0 | Alessio asks for an introductory breakdown of the benchmark architecture. Pratik gives a comprehensive overview of how Galileo aggregated 14 filtered datasets and built the Tool Selection Quality metric. | |
| Surprising Leaderboard Results and Model Rankings | 4 | 4 | 1 | 2 | Alessio highlights surprising findings like Mistral Small's performance, prompting Pratik to share unexpected takeaways like reasoning models failing multi-tool requests. Swyx briefly steps in to clarify a model name mix-up. | |
| Comparative Evaluation Datasets and the Role of Tau-bench | 7 | 4 | 0 | 1 | Swyx demonstrates up-to-the-minute domain knowledge by informing Pratik of Sierra's brand-new Tau-squared release and its telecom domain addition. Pratik explains the value of Tau-bench's deterministic database verification. | |
| Validating LLM-as-a-Judge and the Tool Selection Quality Metric | 5 | 5 | 1 | 2 | Swyx presses on calibration errors and industry skepticism around LLM-as-a-judge approaches. Pratik responds with a detailed explanation of their prompt engineering rigor, AUROC correlation validation, and majority voting. | |
| Prompt Engineering and Model Selection for Evaluator Judges | 4 | 4 | 0 | 0 | Alessio asks about prompt sensitivity and ranking shifts during judge model evaluation. Pratik explains the team's prompt refinement workflow and emphasizes the necessity of using frontier models like Claude 3.7 to judge difficult outputs. | |
| Designing Agent Leaderboard V2 with Multi-Turn Simulation | 2 | 6 | 0 | 0 | Swyx invites Pratik to outline Leaderboard V2. Pratik delivers a detailed breakdown of user simulation, synthetic tool response generation via GPT-4.1 mini, and the introduction of the Action Completion metric to avoid score saturation. |
Statements from this episode (12)
General LLM benchmark rankings do not translate to agent performance
“What we have understood from our previous experiments is that it's not necessary that what you see as top models, In let's say LM arena or other specific evaluation or general evaluations, they might not be also the same ranking for other tasks like agentic ta…”
Top LLMs hold marginal performance edges over cheaper tiers
“Historically, what we have seen from our previous initiatives is that, okay, maybe the best GPT or best cloud is the top model, but there might be very small gap with the model just below it. And then it becomes a cost performance trade off so that users can k…”
Google Gemini Flash-Lite ranks in top five on Galileo Agent Leaderboard
“In the, on the original leaderboard, we also have flashlight, which is, nobody talks about, I think, even now, but I think it's got a very decent score, which is, like, in the top five models. It's extremely cheap model, like, it's so dirt cheap that it makes …”
OpenAI o-series reasoning models fail at multi-tool calling benchmarks
“Then another surprise for me was that the reasoning models were not performing well enough. They had certain kind of limitation when we probed into it, like, why are they scoring less overall? They were like the O-one, the O-four, O-three, they, When not perfo…”
Meta Llama 3.3 and Llama 4 perform poorly on agent benchmarks
“Another, of course, the other surprise was that all the Lama models were not performing well on our benchmark. 3.3 and even the Lama four all were really performing extremely poor.”
Claude 3.7 remains unbeaten on Galileo Agent Leaderboard
“When we released the leaderboard and just in a week that launched 3.7, And that went straight up, and nobody has beaten it so far.”
LLM-as-a-judge methodology is now reliable enough for production evaluation
“Two years back where we were using our own similar LLMS judge methodology, and we had certain issues and there were some correlation issues at the time, but last year it got pretty strong and this year I feel it's so strong. LLMS judges is so strong that you c…”
Changing LLM judge models can heavily alter benchmark leaderboards
“The leaderboard, if you are using different judges with different models, it can, there can be heavy shakeup of the leaderboard also.”
GPT-4o Mini cannot accurately evaluate complex Claude 3.7 outputs
“GPD for a mini can't find, ah, cannot evaluate the hard outputs that's 3.7 might be doing correctly, right?”
Galileo Agent Leaderboard top scores have saturated at 0.95
“Then we saw with V-one that the scores are saturating. We see that the best score has already reached .95.”
Berkeley Function Calling Leaderboard is unsuitable for realistic agent evaluation
“This BFCL if you just open and check it you would be like, okay, this is okay, but you don't want to use it for realistic evaluation because it's like some math. Problem statement is given. Some simple tool is given just like, let's say some exponential tool, …”
Frontier LLMs remain unreliable at realistic multi-turn tool calling
“Our last leaderboard is saying that models are really great at tool calling. So it's like safe, but they actually not, right? They're making mistakes and this is going to recalibrate the expectation of the users that Be careful because they're still not perfec…”