Sridhar: High HLE benchmark scores do not translate to deep research products
Mukund Sridhar · Why is everyone cloning Deep Research? · Feb 18, 2025 · at 50:03
Mukund Sridhar is the tech lead for Gemini Deep Research at Google. He explains why Google avoids over-optimizing research agents for synthetic benchmark tests.
0:00 / 0:21exact quote · 21.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The benchmarks, at least the ones that we are seeing, they don't directly translate to the product. There's definitely some technical challenges that you can benchmark against, but they don't really, like if I do grade on HLE, that doesn't really mean I'm a great deep researcher, so we want to avoid going into that rabbit hole a bit”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mukund Sridhar
Sridhar: Vector dot-product RAG breaks down on multi-attribute queries
“The tricky thing for RAG, it really works well because a lot of these things are doing like cosine distance, like a dot product kind of a thing, and that kind of gets challenging when your query side has multiple different attributes. The dot product doesn't r…”
Sridhar: Horizontal plug-and-play AI agent platforms are premature
“I feel like it's still early days for us, like to try to platformatize or like try to build these, oh, there are these five horizontal pieces. And you can plug and play and build your own agent. My personal opinion is we are not there yet. In order to build a …”
Sridhar: Gemini Deep Research falls back to RAG beyond context limits
“We also have we have retrieval mechanisms, if required. So we natively try to use the context as much as it's available beyond which you know, we have, like, a rag setup to figure out”
Sridhar: Deep Research targets multi-tab exploratory queries rather than direct searches
“There are things that, you know exactly what you're looking for and their search is still probably, you know, a very, you know, probably one of the best places to go. I think where deep research really shines is that, like, Multiple facets to your question, an…”
Sridhar: Sequential grounding on prior search turns is key for deep research
“This notion of being able to read outputs from the previous turn ground on that to decide what to do next, I think was key. Otherwise, you have, like, incomplete information, and your report becomes a little bit of a, like, a high-level bullet point.”