Wu: AI autonomous task duration grew from 10 seconds to 18 hours
Scott Wu · How AI Agents Are Creating 12X Productivity Gains · Joe Lonsdale · Mar 27, 2026 · at 44:54
Scott Wu is the co-founder of Cognition (creator of Devin). He discusses model benchmark progression measured by METR with host Joe Lonsdale.
0:00 / 0:35exact quote · 35.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“One of the stats that people talk about a lot is this METR report, which basically says for each different model that comes out roughly how much human work can it do in an automated fashion before you have to go interrupt it and say, oh, that was wrong. Let's go do this. Right. And so it's like, you know, just two or three years ago, the answer was like 10 seconds or something. You know, you would have it write one line and then it's like, all right, the next line is already wrong. Let's stop here. Right. And at this point, it's already gotten to the point where it's in the scale of, you know, 10:20 hours. Is what the latest one has been. I think Opus 4.6, for example, I think was around 18 hours.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Scott Wu
Wu: Reinforcement learning will eventually beat any benchmark with a clear feedback loop
“I think the natural conclusion of RL, which is what we're kind of getting to, is you basically can solve any benchmark, which is insane to think about... Which means like, if you have a clean set of environments, if you have a good feedback loop to decide what…”
Wu: Devin routinely delivers 6x to 20x efficiency gains for enterprise customers
“We pretty routinely see spots where we can make things like anywhere from six X to 20 X more efficient.”
Wu: Within five years, software engineering will just be describing what to build
“And I think that's kind of what we're going to get to over the next, I'll call it like five or so years is a point where Software engineering. We might even still call it software engineering or programming or whatever, but it's, it really is just you telling …”
Wu: Students should still study computer science despite AI coding advances
“People ask me all the time, you know, it's my son or daughter is, 16, you know, should they even be studying computer science? And I always say, yeah, absolutely. And I think the reason is because these are, you know, the concept level things that actually all…”
Wu: Lowering barrier to coding will drastically increase total software built
“And I think we're going to see something similar here where it's, you know, imagine, imagine the w w w what we will do, you know, when we get to the point where everyone can just build products and websites and apps and whatever it is of that level of quality …”