Galileo Agent Leaderboard top scores have saturated at 0.95
Pratik Bhavsar · ⚡️Ranking Agentic LLMs — Pratik Bhavsar, Galileo · Jul 14, 2025 · at 25:17
Pratik Bhavsar of Galileo explains why they designed Agent Leaderboard V2 to evaluate multi-turn LLM agent performance on harder, domain-specific tasks.
0:00 / 0:07exact quote · 7.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Then we saw with V-one that the scores are saturating. We see that the best score has already reached .95.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Pratik Bhavsar
OpenAI o-series reasoning models fail at multi-tool calling benchmarks
“Then another surprise for me was that the reasoning models were not performing well enough. They had certain kind of limitation when we probed into it, like, why are they scoring less overall? They were like the O-one, the O-four, O-three, they, When not perfo…”
Meta Llama 3.3 and Llama 4 perform poorly on agent benchmarks
“Another, of course, the other surprise was that all the Lama models were not performing well on our benchmark. 3.3 and even the Lama four all were really performing extremely poor.”
LLM-as-a-judge methodology is now reliable enough for production evaluation
“Two years back where we were using our own similar LLMS judge methodology, and we had certain issues and there were some correlation issues at the time, but last year it got pretty strong and this year I feel it's so strong. LLMS judges is so strong that you c…”
GPT-4o Mini cannot accurately evaluate complex Claude 3.7 outputs
“GPD for a mini can't find, ah, cannot evaluate the hard outputs that's 3.7 might be doing correctly, right?”
Frontier LLMs remain unreliable at realistic multi-turn tool calling
“Our last leaderboard is saying that models are really great at tool calling. So it's like safe, but they actually not, right? They're making mistakes and this is going to recalibrate the expectation of the users that Be careful because they're still not perfec…”