The Ledger, every show

“For it to be a business, you need to align with the problem, not the technology. And I think that Automatic optimization is a really great business problem to solve. And I think if you're too fixated on fine tuning as the solution to that problem, then you're …”

“There's a lot of cases now, especially with large context models, where in context learning just beats fine tuning.”

“Pre-Claude III, it was close to a hundred percent OpenAI.”

“I don't have any insider information, so I don't know about the hardware companies, but I do know for some of this, excuse me, for some of the software companies, they have high margins and they're making money.”

“I would say about 50% of the use cases that we see are what I would call like single prompt manipulations.”

“I'd say like probably 25% of the remaining usage is what you could call like a simple agent. Which is probably, you know, a prompt plus some tools. At least one or perhaps the only tool is a rag type of tool, and it is kind of like an enhanced, you know, chatb…”

“If you build your system in a way that Kind of assumes LLMs will get better at reasoning and get better at sort of agentic tasks in the LLM itself. Then I think you will build a more durable system.”

“My conclusion is that this is a very real problem for a very small number of companies, and that is why Datadog, Splunk, Honeycomb, et cetera, built their own database technology, which is, in some ways, it's sad because all of the technology is a remix of pie…”

“And I think the way that enterprises will collect data and leverage it into, you know, these AI processes does not look like doing ETL on a data warehouse that's, you know, running in, in Amazon or something like that. I think it's gonna totally change.”

“Relational databases are perfectly capable of adding HNSW indices to them.”

“Actually, for example, if you do a front-end interview at Braintrust, one of the questions involves writing some C++, and we lose a lot of candidates because of that question but it's a good signal that maybe Braintrust isn't the right place for you to work.”

“Another thing that we're really bullish on at Braintrust is people being in the office and being really comfortable being interrupt-driven.”

“Offline evals, which I think is what a lot of the debate was about are both the most challenging to build feedback loop and also the most efficient once built. And then I think AB tests are a little bit less challenging to build and a little bit less efficient…”

“The first is it is the use case that has the most product market fit in AI, I think, other than ChatGPT.”

“Like if you construct evals in a way that represent the true north star of the problem that you're solving, then they tend not to break as you change the underlying system. Whereas if you hard code them to a narrow subset of like an implementation detail of yo…”

“The interesting thing is that most customers, or actually I'd say a stronger statement, like all customers are quite afraid and reticent to just hand over the data that they use to do evals on to labs.”

“I think you have to pay attention to literally every possible sign that the software is not working. And I think the first step is actually just believing that when you type something or you build a feature, it's not going to work.”

“Really good AI people are able to intuit what the quality curve will look like for a new feature.”

“I think recruiting is one of those sort of trades that you need to learn by working as an apprentice with someone who's really good at recruiting.”

“Raising money was no indication of the quality of your idea or product market fit or chances for finding product market fit.”

“I think that sales or go-to-market founders often have this intuition earlier than engineering founders do is that markets and what people actually want are in many ways out of your control.”

“I think that people without us having to force them to or ask them to, they are just interested in solving the problem around evals and they find it intellectually fascinating on their own. And I think that leads to a lot of organic knowledge sharing. A lot of…”

“There's one library called Tantivi, which is a Rust re-implementation of the Lucene search index, which is a very popular and well-regarded and used search technology. It's the technology behind Elasticsearch as well. And that was the only piece of code that w…”

“We say our product is for product engineering teams that are trying to incorporate AI into their core products and services. And that description excludes and excluded a lot of, ah, sort of traditional enterprises, for example, early on, and we were okay with …”

“So I cut off meetings at noon every day. I don't take any meetings afternoon.”

“I work every weekend. And I have since I joined MemSQL.”

“DuckDB has a struct type, which is dynamically constructed, but it has all the downsides of traditional structured data types, right? So it's just not Like, for example, if you create, if you infer a bunch of rows with the struct type, and then you present the…”

“At Impira, I took kind of the popular advice, which is that developers are a terrible market. So we sold to line of business, and there are a number of benefits to that. Like, we were able to sell six- or seven-figure deals much more easily than We could at Si…”

“However, I learned firsthand that if you don't have a very deep, intuitive understanding of your customer, everything becomes harder. Like, you need to throw product managers at the problem. Your own ability to see around corners is much weaker. And, you know,…”

“And three, Figma is kind of like Apple, a company that is really optimized around a periodic, like annual release cycle rather than something that's continuous.”

“And I think in the absence of evals, what I saw at Impera, and I see with almost all of our Customers before they start using brain trust is this kind of like stalemate between people on which prompt to use or which model to use or which technique to use that …”

“Our core belief is that if you embrace evaluation as The sort of core workflow in AI engineering, meaning every time you make a change, you evaluate it, and you use that to drive the next set of changes that you make, then you're able to build much, much bette…”

“I think it kind of represents the future of AI engineering, one where You can spend a lot of time writing English and sort of crafting the use case itself. You can reuse tools across different use cases. And then most importantly, the development process is ve…”

“Zapier was our first user, and then Coda and Airtable quickly followed, and there was just no chance they would be able to use the product unless the data stayed in their cloud.”

“Now I would say every customer and probably north of 75% of the users that are running evals in brain trust are using the TypeScript SDK. It's an overwhelming majority.”

“At the time, and still, like, AI is sort of at least nominally dominated by Python, but product building is dominated by TypeScript.”

“So in my observation, database companies tend to succeed when the storage paradigm Is closely tied to the execution paradigm, and both of those things need to be rewired to work. I think, remember, the databases are not just storage, but they're also compilers…”

“Unambiguously, people are doing rag. So that one is, you know, it's like simple and obvious. Probably around 50% of the use cases that we see in production involve rag of some sort.”

“Especially, you know, Claude III-V Sonnet has really taken off.”

“Well, I went through this myself watching the technology that we built to do document extraction at Impura become, you know, totally irrelevant.”

“There's some companies that we talked to and their AI vendors are, it's literally OpenAI, AWS, and Braintrust and pretty much everything else has consolidated away.”

“In AI, the whole point of observability is to collect data into data sets that you can use to do evals, and then again, eventually fine tune models or, you know, more advanced things.”

“It's way easier for an LLM, especially a frontier model, to look at the work of you know, itself or another LLM and accurately assess it.”

“We've seen like a massive surge of product manager, product managers and designers getting interested in participating in the eval process among our customers.”

“It's pretty easy, and now I would say even like easier to build a prototype, but the craft of actually taking something from 95% to 99% or a hundred percent is very challenging and not something they teach you in school.”

“Recruiting really, really good engineers is not a transactional process. I think that sounds very obvious to anyone who's tried or read about recruiting, but until you experience it, it's very hard. We sometimes spend years recruiting someone”

“For example, I was at Figma recently, I think they're really exceptional at it.”

“We had a few million dollars in ARR, but very little repeatability across what they were doing. So when we tried to bring on a sales team and scale that what they were selling and their ability to scale the success from those early users into the end plus firs…”