Sehgal: Deep research evals should categorize research behavior, not domain verticals
Arush Sehgal · Why is everyone cloning Deep Research? · Feb 18, 2025 · at 24:53
Arush Sehgal (Lead PM for Gemini Deep Research) explains how Google developed their evaluation benchmark ontology across research archetypes instead of specific industries.
0:00 / 0:13exact quote · 14.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“And really what we tried to do is like, stay away from like verticals, like travel or shopping and things like that, but really try and go into like, what is the underlying research behavior Type that a person is doing.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Arush Sehgal
Sehgal: Keep recent research tasks in context, relegating older ones to RAG
“Just to add to that, I think like, just like a simple rule of thumb that we use is like, if it's the most recent set of research tasks where the user is likely to ask lots of follow-up questions, that should be in context. But like, as stuff gets 10 tasks ago,…”
Sehgal: AI deep research provides most lift on niche, non-Wikipedia topics
“We love to test, like, super niche random things, like, things where there's, like, No Wikipedia page already about this topic or something like that, right? Because that's where you'll see the most lift from a feature like this.”
Sehgal: Google shipped a 5-minute Deep Research mode fearing user drop-off
“I remember we actually built two versions of deep research. We had like a hardcore mode that takes like 15 minutes. And then what we actually shipped is a thing that takes five minutes. And I even went to Eng and I was like, there has to be a hard stop, by the…”
Sehgal: Users always max out AI power toggles if given the option
“If you like give a max power button, users are always just going to hit that button, right? So then the question comes like, why don't you just decide from the product POV where's the right balance?”