Goyal: Pioneering AI companies are abandoning free-form autonomous agents
Ankur Goyal · No Priors Ep. 85 | CEO of Braintrust Ankur Goyal · Oct 8, 2024 · at 18:23
Braintrust CEO Ankur Goyal discusses production architecture patterns with Elad Gil, explaining why engineering teams are stepping back from open-ended autonomous agent loops.
0:00 / 0:42exact quote · 42.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Probably the most consistent thing I've seen is companies kind of walking back from the illusion that totally free form agents will solve all of their problems. So I think maybe like two or three months ago, Many of the pioneering companies went way down the agent rabbit hole. And they kind of realized like, wow, this is actually not, this is not, not the right approach. It's so hard to control performance. The error rates are really high and they compound really quickly. And so, you know, most of those companies have kind of walked back and tried to build a different architecture where the control flow is, is actually managed deterministically by their code.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ankur Goyal
Goyal: TypeScript's type system makes it better suited for AI than Python
“Another thing is that TypeScript as a language is inherently better suited for AI workloads because of the type system. So the type system basically allows you to launder, you know, the crazy stuff that comes out of an AI model into a well-defined structure th…”
Goyal: Nearly all Braintrust customers have abandoned fine-tuned models
“Almost if not all of our customers have moved off of fine-tuned models onto instruction-tuned models and are seeing really good performance.”
Goyal: Practical adoption of open-source models remains very limited
“So we see very limited practical adoption of open source models, but I think more interest than ever.”
Goyal: Open-source models will struggle until they improve UX or iteration speed
“So I think until open source can really move the needle on one of those two axes, it's going to be tough for it to be adopted broadly.”
Goyal: Internet-trained LLMs outperform models trained on internal enterprise data
“And I think the big insight or the crazy, you know, non-intuitive thing about LLMs is that something trained on the internet outperforms what an enterprise can produce with their own data trained on data in a data warehouse.”