Opinion
Goyal: TypeScript's type system makes it better suited for AI than Python
“Another thing is that TypeScript as a language is inherently better suited for AI workloads because of the type system. So the type system basically allows you to launder, you know, the crazy stuff that comes out of an AI model into a well-defined structure th…”
Assertion Not checkable as stated
Goyal: Nearly all Braintrust customers have abandoned fine-tuned models
“Almost if not all of our customers have moved off of fine-tuned models onto instruction-tuned models and are seeing really good performance.”
Assertion Not checkable as stated
Goyal: Practical adoption of open-source models remains very limited
“So we see very limited practical adoption of open source models, but I think more interest than ever.”
Insight
Goyal: Open-source models will struggle until they improve UX or iteration speed
“So I think until open source can really move the needle on one of those two axes, it's going to be tough for it to be adopted broadly.”
Insight
Goyal: Internet-trained LLMs outperform models trained on internal enterprise data
“And I think the big insight or the crazy, you know, non-intuitive thing about LLMs is that something trained on the internet outperforms what an enterprise can produce with their own data trained on data in a data warehouse.”
Prediction Not checkable as stated
Goyal: Embeddings and LLMs will replace relational indexes for querying data
“LLMs and, you know, specifically embeddings are going to be core to how people actually query data, not, you know, traditional algebraic relational indexes.”
Prediction Not checkable as stated
Goyal: AI semantic search will disrupt OLAP far more than OLTP
“What will really be disrupted is the OLAP workload. So relational, you can't just slap you know, semantic search and stuff into the architecture of a traditional data warehouse. I think that is actually a much deeper set of things that will need to change than…”
Insight
Goyal: Pioneering AI companies are abandoning free-form autonomous agents
“Probably the most consistent thing I've seen is companies kind of walking back from the illusion that totally free form agents will solve all of their problems. So I think maybe like two or three months ago, Many of the pioneering companies went way down the a…”
Disclosure
Goyal: Vast majority of Braintrust customers now use TypeScript over Python
“First of all a vast majority of our customers use TypeScript and, you know, early on, some of our customers were dealing with, like, should we use TypeScript or Python? And some teams were using TypeScript, some teams were using Python. Now, almost everyone, i…”
Insight
Goyal: Software engineering teams are abandoning specialized AI application frameworks
“The biggest thing I've seen over the past six months is, People dropping the use of frameworks.”
Opinion
Goyal: AWS regained its mojo by hosting Anthropic Claude on Bedrock
“AWS has its mojo back now that they have Anthropic on bedrock and Anthropic is, you know, especially cloud three and three, five are really, really good.”
Prediction Not checkable as stated
Goyal: Enterprise AI data infrastructure will move away from data warehouse ETL
“And I think the way that enterprises will collect data and leverage it into, you know, these AI processes does not look like doing ETL on a data warehouse that's, you know, running in, in Amazon or something like that. I think it's gonna totally change.”
Assertion Supported
Goyal: Relational databases are fully capable of adding HNSW vector indices
“Relational databases are perfectly capable of adding HNSW indices to them.”
Disclosure
Goyal: Braintrust requires front-end engineering candidates to write C++
“Actually, for example, if you do a front-end interview at Braintrust, one of the questions involves writing some C++, and we lose a lot of candidates because of that question but it's a good signal that maybe Braintrust isn't the right place for you to work.”
Disclosure
Goyal: Braintrust embraces in-office work and an interrupt-driven engineering culture
“Another thing that we're really bullish on at Braintrust is people being in the office and being really comfortable being interrupt-driven.”
Assertion Not checkable as stated
Goyal: 50% of enterprise AI production use cases involve RAG
“Unambiguously, people are doing rag. So that one is, you know, it's like simple and obvious. Probably around 50% of the use cases that we see in production involve rag of some sort.”
Assertion Not checkable as stated
Goyal: Anthropic's Claude 3.5 Sonnet has really taken off
“Especially, you know, Claude III-V Sonnet has really taken off.”
Assertion Not checkable as stated
Goyal: Impira's document extraction tech became totally irrelevant with LLMs
“Well, I went through this myself watching the technology that we built to do document extraction at Impura become, you know, totally irrelevant.”
Assertion Not checkable as stated
Goyal: Some enterprises consolidated AI stacks to OpenAI, AWS, and Braintrust
“There's some companies that we talked to and their AI vendors are, it's literally OpenAI, AWS, and Braintrust and pretty much everything else has consolidated away.”
Insight
Goyal: AI observability exists to collect datasets for evaluations and fine-tuning
“In AI, the whole point of observability is to collect data into data sets that you can use to do evals, and then again, eventually fine tune models or, you know, more advanced things.”
Insight
Goyal: Validating LLM outputs is significantly easier for frontier models than generation
“It's way easier for an LLM, especially a frontier model, to look at the work of you know, itself or another LLM and accurately assess it.”
Disclosure
Goyal: Braintrust's early growth came from targeting 50 key AI innovators
“Really the thing that we did was we made that list of, like, 50 people who we thought were leading the way in AI and said, you know, let's try to figure out a way to get to these people and either get, recruit them as investors or as customers. And I think tha…”
Assertion Not checkable as stated
Goyal: Over half of evaluations run on Braintrust are LLM-based
“I think probably more than half of the evals that people do in Braintrust are LLM based.”
Ankur Goyal