Graham Neubig
9 statements across 2 episodes · 4 bullish · 4 bearish · 2 people on the record · first statement Dec 25, 2024 by Graham Neubig · said 10 times in 4 episodes since 2024 · across every show →
On the record as a speaker too: Graham Neubig's record, appearances and statements → this page counts the times other people say the name.
Mentions by year
brought up most by Shawn Wang (5), Swyx (Marcos Swix) (4)
tap a year for its mentions
2025 8 mentions in 2 episodes 4 per episode
2024 2 mentions in 2 episodes 1 per episode
Everything said about Graham Neubig, oldest first
Dec 25, 2024 negative
Dec 25, 2024 bearish
Neubig: SWE-bench Scores Are Inflated By Training Data Contamination
“Sweebench is on popular open source repos and all of these popular open source repos were included in the training data for all of the language models. And so, the language models already know these repos. In some cases, the language models already know the in…”
Dec 25, 2024 positive
Neubig: Agents Solve 80 To 90 Percent Of Tasks With Feedback
“Actual ability of models is maybe closer to 30 to 40%. So 30 to 40% of the things that I want an agent to solve on my own repos, it just solves without any human intervention. 80 to 90% it can solve without me opening an IDE, but I need to give it feedback.”
Dec 25, 2024
Dec 25, 2024 positive
Dec 25, 2024 bullish
Dec 25, 2024 bearish
Neubig: Most APIs Lack The Fine-Grained Authentication Needed For Agents
“For other things, they're totally not prepared to give that sort of fine-grained control. Like, most APIs don't have something like a fine-grained authentication token, and that goes into my, like, comment that we're gonna need to prepare the world for agents,…”
Dec 25, 2024 negative
Neubig: Inadequate Information Gathering Is The Biggest Agent Failure Mode
“So I think actually probably the biggest thing that it fails at is. Or that our agent plus Claude fails at is insufficient information gathering before trying to solve the task, and so if you provide all, if you provide instructions that it should do informati…”