Oct 8, 2024 · 38m · no-priors
No Priors Ep. 85 | CEO of Braintrust Ankur Goyal
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of No Priors, host Elad Gil interviews Ankur Goyal, founder and CEO of Braintrust, to explore the evolution of enterprise AI evaluation, data infrastructure shifts, and production engineering practices. Goyal explains why enterprises are converging on deterministic control flows, TypeScript, and proprietary frontier models while outlining Braintrust's journey toward becoming a universal platform for AI development.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 20.2% of the talking time here. How this is scored →
The hosts as informed peer 4.4 Guest teaching 5.3 Guest disagreement 1.7 The hosts pushing back 1.0
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 24:05 TypeScript superior to Python for AI systems
Ankur takes a firm contrarian stance against Python advocates, highlighting that despite getting trolled on Twitter, TypeScript is objectively better suited for uncertain AI data shapes.
Hardest push from the hosts ▶ 20:33 Elad probes ML resistance motivesElad challenges whether ML practitioner skepticism of LLMs was really due to technical problem-set mismatches rather than general dismissal, prompting Ankur to explain the emotional disruption.
Biggest teaching moment ▶ 14:00 Debunking vector DB hype and explaining OLAP disruptionAnkur dismisses the popular discourse around vector databases as silly and educates the audience on why semantic search fundamentally breaks traditional relational OLAP data warehouse architectures.
The host holds their own ▶ 29:35 Elad connects Jensen Huang's CEO architecture to startup executionElad synthesizes Jensen Huang's management thesis on structuring companies around founder strengths with standard operational traps like reinventing sales compensation.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Genesis of Braintrust: Lessons from Impira to Figma | 5 | 4 | 1 | 1 | Elad shares his early personal observations from customer calls with Braintrust and framing why customers demanded commercial software over open source. Ankur explains why evals are deceptive and how early pioneers realized it was harder than a simple for loop. | |
| Enterprise Adoption Trends: RAG vs. Fine-Tuning | 4 | 6 | 2 | 1 | Ankur reframes the debate around fine-tuning, explaining that fine-tuning is merely a technique while automatic optimization is the real outcome, and points out that nearly all production customers have migrated back to instruction tuning. | |
| Open Source vs. Proprietary Model Economics in Production | 5 | 5 | 2 | 1 | Elad contextually grounds Ankur's insights as reflecting live production systems at scale rather than hobbyist benchmarks. Ankur details the economic realities of proprietary APIs beating open-source self-hosting on developer velocity and ROI. | |
| Embeddings, Semantic Search, and Analytical Workload Disruption | 4 | 7 | 2 | 1 | Drawing on his database background from SingleStore, Ankur delivers deep insights dismissing the standalone vector database debate as trivial while arguing that semantic search and embeddings will completely disrupt traditional OLAP warehouse workloads. | |
| Architectural Realism: Moving from Autonomous Agents to Deterministic Code | 5 | 5 | 2 | 1 | Elad outlines the typical journey enterprise startups make before reaching maturity. Ankur validates this by breaking down how pioneering companies walked back from autonomous agent while-loops in favor of deterministic code with LLM calls throughout. | |
| Evolution of AI Teams and the Dominance of TypeScript | 5 | 6 | 3 | 1 | Elad probes into why traditional ML engineers resisted LLMs, and Ankur offers an honest psychological explanation before making a bold contrarian case for TypeScript dominating AI over Python due to strong typing and product engineering dominance. | |
| Tooling Shifts: Framework Abandonment and Hyperscaler Consolidation | 5 | 5 | 1 | 1 | Ankur points out how framework usage has been abandoned in favor of pervasive native software engineering alongside massive vendor consolidation around AWS and OpenAI. Elad draws comparisons to other coding CEOs like Tobi Lutke and Jensen Huang. | |
| Operational Edge: In-Person Work, Fast Feedback Loops, and GTM Strategy | 4 | 4 | 1 | 1 | Ankur discusses their strict in-office requirement, interrupt-driven fast iteration speed, and high-touch GTM strategy. Elad adds his perspective on how startups commonly misdefine their initial customer envelopes. | |
| Future of Braintrust: Universal Developer Platform and Automated Evals | 3 | 6 | 1 | 1 | Ankur explains how customer pull transformed Braintrust from pure evals into an observability and prompt engineering platform, and outlines how LLMs are increasingly used to evaluate logs containing private PII. |
Statements from this episode (24)
Gil: Early Enterprise Prospects Asked Braintrust Not to Open Source
“I remember in the early conversations we had around the company or the idea, I should say, it was meant to even potentially be open source. And as the first time that I was involved with some sort of customer call and people would say, we don't want you to ope…”
Goyal: 50% of enterprise AI production use cases involve RAG
“Unambiguously, people are doing rag. So that one is, you know, it's like simple and obvious. Probably around 50% of the use cases that we see in production involve rag of some sort.”
Goyal: Nearly all Braintrust customers have abandoned fine-tuned models
“Almost if not all of our customers have moved off of fine-tuned models onto instruction-tuned models and are seeing really good performance.”
Goyal: Anthropic's Claude 3.5 Sonnet has really taken off
“Especially, you know, Claude III-V Sonnet has really taken off.”
Goyal: Practical adoption of open-source models remains very limited
“So we see very limited practical adoption of open source models, but I think more interest than ever.”
Goyal: Open-source models will struggle until they improve UX or iteration speed
“So I think until open source can really move the needle on one of those two axes, it's going to be tough for it to be adopted broadly.”
Goyal: Internet-trained LLMs outperform models trained on internal enterprise data
“And I think the big insight or the crazy, you know, non-intuitive thing about LLMs is that something trained on the internet outperforms what an enterprise can produce with their own data trained on data in a data warehouse.”
Goyal: Enterprise AI data infrastructure will move away from data warehouse ETL
“And I think the way that enterprises will collect data and leverage it into, you know, these AI processes does not look like doing ETL on a data warehouse that's, you know, running in, in Amazon or something like that. I think it's gonna totally change.”
Goyal: Embeddings and LLMs will replace relational indexes for querying data
“LLMs and, you know, specifically embeddings are going to be core to how people actually query data, not, you know, traditional algebraic relational indexes.”
Goyal: Relational databases are fully capable of adding HNSW vector indices
“Relational databases are perfectly capable of adding HNSW indices to them.”
Goyal: AI semantic search will disrupt OLAP far more than OLTP
“What will really be disrupted is the OLAP workload. So relational, you can't just slap you know, semantic search and stuff into the architecture of a traditional data warehouse. I think that is actually a much deeper set of things that will need to change than…”
Goyal: Braintrust requires front-end engineering candidates to write C++
“Actually, for example, if you do a front-end interview at Braintrust, one of the questions involves writing some C++, and we lose a lot of candidates because of that question but it's a good signal that maybe Braintrust isn't the right place for you to work.”
Goyal: Pioneering AI companies are abandoning free-form autonomous agents
“Probably the most consistent thing I've seen is companies kind of walking back from the illusion that totally free form agents will solve all of their problems. So I think maybe like two or three months ago, Many of the pioneering companies went way down the a…”
Goyal: Impira's document extraction tech became totally irrelevant with LLMs
“Well, I went through this myself watching the technology that we built to do document extraction at Impura become, you know, totally irrelevant.”
Goyal: Vast majority of Braintrust customers now use TypeScript over Python
“First of all a vast majority of our customers use TypeScript and, you know, early on, some of our customers were dealing with, like, should we use TypeScript or Python? And some teams were using TypeScript, some teams were using Python. Now, almost everyone, i…”
Goyal: TypeScript's type system makes it better suited for AI than Python
“Another thing is that TypeScript as a language is inherently better suited for AI workloads because of the type system. So the type system basically allows you to launder, you know, the crazy stuff that comes out of an AI model into a well-defined structure th…”
Goyal: Software engineering teams are abandoning specialized AI application frameworks
“The biggest thing I've seen over the past six months is, People dropping the use of frameworks.”
Goyal: AWS regained its mojo by hosting Anthropic Claude on Bedrock
“AWS has its mojo back now that they have Anthropic on bedrock and Anthropic is, you know, especially cloud three and three, five are really, really good.”
Goyal: Some enterprises consolidated AI stacks to OpenAI, AWS, and Braintrust
“There's some companies that we talked to and their AI vendors are, it's literally OpenAI, AWS, and Braintrust and pretty much everything else has consolidated away.”
Goyal: Braintrust embraces in-office work and an interrupt-driven engineering culture
“Another thing that we're really bullish on at Braintrust is people being in the office and being really comfortable being interrupt-driven.”
Goyal: Braintrust's early growth came from targeting 50 key AI innovators
“Really the thing that we did was we made that list of, like, 50 people who we thought were leading the way in AI and said, you know, let's try to figure out a way to get to these people and either get, recruit them as investors or as customers. And I think tha…”
Goyal: AI observability exists to collect datasets for evaluations and fine-tuning
“In AI, the whole point of observability is to collect data into data sets that you can use to do evals, and then again, eventually fine tune models or, you know, more advanced things.”