Howard: Autonomous AI coding agent Devin produces low-quality, useless software
Jeremy Howard · Jeremy Howard on Building 5,000 AI Products with 14 People (Answer AI Deep-Dive) · May 15, 2025 · at 31:41
Jeremy Howard, co-founder of Answer.ai, details Answer.ai's hands-on testing of Cognition AI's autonomous coding agent Devin.
0:00 / 0:39exact quote · 39.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Devon's an agent you know, a bunch of tools, tool calls, and tied together with an OpenAI model, if I understand correctly. Yeah, and it actually ended up definitely supporting our thesis, which is, there are so many places we wished we could have got involved, you know, and could have, like, worked together and be like, I know how to debug that, or, like, it's going on wild tangents, or, you know, it ended up taking a lot longer to do things than we would do them on our own, and ended up with much less high-quality software, and it ended up with kind of vibe-coded stuff that we couldn't really build on top of anyway, which kind of, in a lot of situations, it has no value, you know.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jeremy Howard
Howard: Closed US AI ecosystems will cause China to move faster
“Whereas, oddly, the US companies that used to lead the way have all drawn up the drawbridges, and that's gonna cause China to keep moving faster, because when you're in that more, both collaborative and competitive environment, you just Go way ahead, as we've …”
Howard: OpenAI compute spending grows exponentially while model utility scales logarithmically
“They kept on kind of exponentially increasing the amount they were spending on their models, whilst the Return, you know, the kind of utility of those models was only increasing logarithmically, and you kind of very quickly hit this point where it's like, oh, …”
Howard: Test-time compute scaling will hit diminishing returns within two years
“It's a thing where you get most of the juice out of it in the first year or two, so we're still in that. Phase at the moment, and we'll start to hit the curve off point pretty soon. Just like we did for training.”
Howard: No more evidence for near-term ASI today than 15 years ago
“I don't think we have any more evidence that ASI might be close now than we did 15 years ago, 15 years before that.”
Howard: AI growth will continue tailing off as low-hanging fruit disappears
“The amount of people and money being put into AI, it's going to keep going up, but again, the low hanging fruits being done now. So yeah, I think we'll continue to see the tailing off of growth.”
Howard: Traditional AI labs favor paper citations over bold innovation
“It's very different to a research lab, which is like, hey, let's try and get lots of citations on a paper in a field that's so highly recognizable to its peers that they recognize it as being something they want to appear at their next conference, you know, wh…”