The Ledger, every show

“It's just not practical outside of OpenAI to run use cases at scale in a lot of cases. Like, you can do it, but it requires quite a bit of work. And Because OpenAI is so good at making their models so available, I think they get a lot of credit for the science…”

“It feels very clear to me that this type of logic is going to be built into the model. Anytime there is control flow complexity or uncertainty complexity, I think the history of AI has been to push more and more into the model.”

“And I think O-one is going to do that to agentic frameworks as well. Hey, I think To me, it seems very unlikely that the, you know, you and me sort of like sipping an espresso and thinking about how, like, different personified roles of people should interact …”

“They are nowhere near as reliable as, I mean, every single time I use any of those products and run a benchmark, I find a bug, text the CEO, and they fix something. It's nowhere near where OpenAI is.”

“Another thing is that TypeScript as a language is inherently better suited for AI workloads because of the type system. So the type system basically allows you to launder, you know, the crazy stuff that comes out of an AI model into a well-defined structure th…”

“It's just that the value proposition of publishing an eval is completely orthogonal to the value proposition of building evals in service of building a good product. I think the purpose of publishing benchmarks is marketing, and it's good marketing.”

“In many ways coming to the table of product building with representative examples and criteria that articulate what good versus bad is for a use case is just a more precise and usable form of product management than writing a spec.”

“We actually heard early on from people and allergic, no reaction to that. They were like, Hey, we were using open source stuff and it's super brittle and it breaks all the time. And we just evals suck and observability sucks and we just don't want to deal with…”

“I wanted at Braintrust to, again, in the spirit of skepticism, Build a terrible go-to-market motion early on, but make the product, make Braintrust only successful if the product was so good that despite being grossly incompetent at selling and marketing our p…”

“I think there's only one right answer, which is it's much better to be overhyped and then fail. I mean, obviously within the realms of morality and integrity, but I was, I sort of learned like, I don't want to be an obscure product that no one cares about.”

“A classic example of this is when companies hire salespeople from extremely successful product-led companies, and they expect the salesperson to be good for their product-led company that does not have product market fit. That is not the right You actually wan…”

“The real gap is that software engineers who have a particular way of thinking, a particular set of biases, a particular type of workflow that they run, are going to be the ones who are doing AI engineering, and that the tools that were built for ML are fantast…”

“For probably 80 or 90% of the use cases that we see with people doing this, like very, very simple, I create a prompt, it calls some tools. I can like very ergonomically write the tools, plug into popular services, et cetera, and then just call them kind of li…”

“The problem is that the challenge in deploying vector search has very little to do with vector search itself, and much more to do with the data adjacent to vector search. So, for example, if you are at Figma, the Vector search is not actually the hard problem.…”

“I will say in my own experience with customers as of the recording date today, which is September or something, yeah, very few of our customers are currently fine-tuning models. And I think a very, very small fraction of them are running fine-tuned models in p…”

“We still see an overwhelming majority of customers using OpenAI, but almost everyone is using Anthropic for the, and Sonnet specifically for their side projects, whether it's You know, via cursor or prototypes or whatever.”

“It hasn't been a smooth journey for people to get the capacity on public clouds that they're able to get through, you know, OpenAI directly. I mean, I think a lot of this is changing, catching up, et cetera. But it hasn't been perfectly smooth. And I think the…”

“If you make assumptions about the capabilities of models, and you engineer around them, you're almost, like, guaranteed to be screwed.”

“Among customers running in production, it's less than five percent.”

“Almost everyone that we work with has gone into this model that, that I, that actually exactly what you said, which is sprinkle intelligence everywhere and make it easy to write dumb code”

“Almost if not all of our customers have moved off of fine-tuned models onto instruction-tuned models and are seeing really good performance.”

“So we see very limited practical adoption of open source models, but I think more interest than ever.”

“So I think until open source can really move the needle on one of those two axes, it's going to be tough for it to be adopted broadly.”

“And I think the big insight or the crazy, you know, non-intuitive thing about LLMs is that something trained on the internet outperforms what an enterprise can produce with their own data trained on data in a data warehouse.”

“LLMs and, you know, specifically embeddings are going to be core to how people actually query data, not, you know, traditional algebraic relational indexes.”

“What will really be disrupted is the OLAP workload. So relational, you can't just slap you know, semantic search and stuff into the architecture of a traditional data warehouse. I think that is actually a much deeper set of things that will need to change than…”

“Probably the most consistent thing I've seen is companies kind of walking back from the illusion that totally free form agents will solve all of their problems. So I think maybe like two or three months ago, Many of the pioneering companies went way down the a…”

“First of all a vast majority of our customers use TypeScript and, you know, early on, some of our customers were dealing with, like, should we use TypeScript or Python? And some teams were using TypeScript, some teams were using Python. Now, almost everyone, i…”

“The biggest thing I've seen over the past six months is, People dropping the use of frameworks.”

“AWS has its mojo back now that they have Anthropic on bedrock and Anthropic is, you know, especially cloud three and three, five are really, really good.”

“People don't really want to create golden data sets. It's, I think it's often a wasted effort to the point that you're making. I think the best teams view offline evals as a mechanism of reconciling what they see in production with real users who are using the…”

“You need to have like a pretty specific expertise to design the RL environment in a way that's not vulnerable to reward hacking. And I think that either you'll end up with some fixed number of very well engineered RL environments, or you need to somehow employ…”

“Like if your product isn't selling for some reason, then it's almost always pipeline.”

“And then I did something which I honestly don't think any founder should do. We're talking about product market fit. This is almost a surefire way to prevent yourself from finding product market fit. But I sat down and I said, okay, I want to start a company. …”

“I think in the world of early product building, you actually need to be like ultra paranoid about repeatability. And you need to assume that, you know, give yourself a little bit of credit and assume that you're somewhat charismatic or Entertaining or whatever…”

“Understandable, but staggering to me how bad relational databases are at full text search. Just, they're just awful.”

“Like, we don't have, we have one meeting per week as a company, and that's it.”

“I think personally, my perspective is even if you're trying to hire like a pretty good, well-rounded and, you know, skilled group of people that's even, you know, 40 or 50 people, it's very challenging to get everyone to work on weekends. And so we don't, that…”

“I think if you have to convince people, it's super unlikely that you have product market fit. Whereas if without convincing people, they just find their way to the product and then they've, they're already convinced or they've convinced themselves very quickly…”

“It is, without any question, at least in my experience, the best implementation of semi-structured data and sort of solves the problem of storing it very, very efficiently and querying it efficiently, almost as efficiently as if you specified the schema exactl…”

“And I think every observability product in some sort of platonic ideal would be built on top of Snowflake's variant implementation. And have better performance. It would be cheaper. You know, the customer experience would be better. But, you know, alas, it's j…”

“It is very, very hard to motivate a large organization to prioritize the problem. And so you're always going to be a second or third tier priority.”

“Vectors are very difficult because they're a data inefficient representation, so the vector format in something like Figma Is choose up like many, many, many, many, many more tokens than HTML and JSX. So it's a very difficult medium to just sort of throw into …”

“In my limited experience and working with designers myself, I think designers do not want AI to design things for them. But there's a lot of things that aren't in the traditional designer toolkit that AI can solve. And I think the biggest one is generating cod…”

“If you structure your code so that the same function abstraction that you define to evaluate on equals equals the abstraction that you actually use to run your application, then when you log your application itself, you actually log it in exactly the right for…”

“Cursor is taking AI and making traditional software engineering like insanely good with AI. And we are taking some of the best things about traditional software engineering and bringing them to building AI software.”

“I would say almost all of the products in the space are spreadsheet plus plus, right? Like, you know, here's a script, generates an eval, I look at the cells, you know, whatever, side by side and compare it.”

“In almost all cases, vector search is not a storage or performance bottleneck. And in almost all cases, the vector search involves exactly one query, which is, you know, nearest neighbors.”