The Ledger, every show

“I think AGI is also a religion. It has a problem of evil. We don't have enough intelligence. It has a solution, a deus ex machina. It has the second coming of Christ that AGI, the singularity is going to come. It's going to save humanity because we will now ha…”

“A few of the insights is, like, everyone, frontier model is not good at search. Humans have this natural explore-exploit trade-off, where we kind of understand, like, when to stop doing something. Also, humans are pretty good at, like, forgetting, actually, li…”

“The performance of LLMs is not invariant to how many tokens you use. As you use more and more tokens, the model can pay attention to less, and then also can reason sort of less effectively.”

“I think that, like, this is going to be the dominant paradigm. I actually think that, like, probably purpose-built re-rankers will go away, and the same way that, like, purpose-built, they'll still exist, right? Like, if you're at extreme scale, extreme cost, …”

“My guess is that, like, for code today, it's something like, 90% of queries or 85% of queries can be satisfactorily run with regex. Regex is obviously, like, the dominant pattern used by Google code search, GitHub code search, but you maybe can get, like, 15% …”

“I think, like, there's a few things that I think might be true about retrieval systems in the future. So, like, number one, they just stay in latent space, they don't go back to natural language.”

“I don't think of an LLM as an operating system. I think an LLM is much more like a CPU, right? It's an information processing unit.”

“Focus on the developers. I think that's the beachhead. That's how you win the B to B market. If you win the B to B market with your open source models, Like, you get all of the sort of downstream effects that you want. You know, you don't need to beat you know…”

“Diminishing, they're clearly diminishing marginal returns, right? We're sort of spending 10 X on compute. We're not getting 10 X or better models, at least evidently not yet.”

“It's a, you know, technology is probably as important as the invention of electricity. It will probably, you know, bring about a increase in GDP that is on the order of the industrial revolution or greater.”

“Like it is very possible the poorest people on earth today, or, you know, in 10 years, we'll have access to better healthcare better legal representation you know, better financial services than, like, billionaires have today.”

“Now, of course, for those of you that have actually played with technology, I think it's questionable whether the current state of the art Language models, embedding models, et cetera, will give you the reliability you want from, ah, you know, agents working t…”

“Well, I'm just saying, I think, like, RBAC does seem to be dead, right? Like, if you want to say something is dead, probably RBAC is dead. And like the auth story to me seems like incredibly unsolved and unaddressed by like the existing state of like AI vendor…”

“We've like sort of stress tested like frontier models and their ability to search. And they are not actually that good at searching.”

“I think that the actual graph structure is emergent in the mind of the agent, ah, in the same way it is in the mind of the human. And that's a more powerful graph, because it actually evolved over time.”

“My critique of that would be that if you follow that methodology, you will probably end up building a dating app for middle schoolers, because that just seems to be like the lowest base take of what humans want to some degree.”

“This is what, frankly, most AI startups, any AI stuff that you know of, that you think of today that's doing very well, like what are they fundamentally good at? What is the one thing that they're good at? It is context engineering.”

“Increasingly is the market to be a good LLM provider, the main market seems to be consumer. You're just not that motivated to, like, help developers.”

“I would rather have a model that has a 60,000 context, token context window, that is able to perfectly pay attention to, and perfectly reason over those 60,000 tokens, than a model that's like five million tokens. Like, just as a developer, the former is like …”

“Like, the idea that there's going to be, like, a lot of offline compute and inference under the hood that helps make AI systems continuously self-improve is a sure bet.”

“So far we've still not found that really any AI coding tools are particularly good at rust though.”

“In the same way that we have a memory hierarchy in classic computers, right, we have the CPU, RAM, disk, and network we are also going to have a similar memory hierarchy in language models. And again, it already exists today. We have the actual sort of transfo…”

“Even these, like, needle-in-a-haystack tests, like, are not actually that representative of, like, real-world utility and reliability of long context windows.”

“Distribution is incredibly important as long as, you know, sort of the incumbents can wake up and can catch up. You know, I would not bet against Zuck And a hundred billion dollars of profit per year.”

“Most businesses don't love using closed source models. They want to use open source models. For all kinds of reasons, you know, privacy, security, continuity, cost”

“Updating the weights of the model is not a very good idea because you cannot really deterministically control that. You can fine tune, but what you're going to get the other end, you know, again, you don't really control.”

“We think the capability overhang we have in the models that we already have today, and we will have absolutely in six months is immense.”

“Chroma will always, we are committed to building the ubiquitous open source standard.”

“And we think that both certainly transactional has to be the case because it is a online database. It's gonna sit in the loop of applications. Again, you've already seen demos of this happening tonight. But also to make this technology useful for developers, y…”

“People, there's this, like, you know, big question about, oh, are vector databases gonna replace classic databases? Are these competitive in some way? And I think it's just kind of a dumb question.”

“And so, like, I think pruning is also going to be, like, really, it's already becoming a thing, right? But, like, letting models, like, self-prune their context windows.”

“Most companies are practically apprenticeships. Like every new employee who joins the team, like you spend one to three months, like wrapping them up. All that tested knowledge is not written down.”

“For many years running, Chrome has been the number one used project broadly, but also within communities like LinkChain and Llama Index.”

“And yeah, they're using context caching and that certainly helps, but like their cost and speed, but like isn't helping the context raw problem at all.”

“As much structured information as you can put into your write or your ingestion pipeline, you should. So all of the metadata you can extract, do it at ingestion. All of the chunk rewriting you can do, do it at ingestion. If you really invest in, like, trying t…”

“A lot of those have the problem where, like, either the retriever or the language model has to be frozen, and then, like, the corpus can't change, which most developers don't want to, like, deal with the developer experience around.”

“So I think generating QA pairs is really important for benchmarking your retrieval system, golden dataset. Frankly, it's also the same dataset that you would use to fine tune in many cases. And so, yeah, there's definitely something like very underrated there.”

“Having worked in applied machine learning developer tools now for 10 years, like the returns to a very high quality small label data set are so high. Everybody thinks you have to have like a million examples or whatever. No, actually just like a couple hundred…”

“Silicon Valley has a tendency to be sort of extremely intellectually shallow. This is both a strength and a weakness of the Valley, to be clear.”

“You know, you see a lot of like model drops that come out, but they don't actually provide the real hooks and they do very well in the benchmarks, right? They do very well on like kind of the public leaderboards. But they don't actually provide the hooks that …”

“I think like today, 90 plus percent of it in enterprises is retrieval event generation, or it's, you know, using retrieval, it's sort of a chat on top of unstructured data.”

“By looking at embeddings doing embedding search, doing analytics over embedding space you could give developers you know, at the minimum of a divining rod, if not a compass, to be able to improve their models and get to the level of reliability they want to ha…”

“Chroma's belief is that programmable memory, so developers being able to set terministically Hey, language model, this is the knowledge you should know about, this is the knowledge you should use, these are the tools you should know about, these are the tools …”

“So I think that use case, even just the Chat Your Data use case, truly will go to the ends of the earth.”

“So, there's lots of stuff around steering language models at the embedding layer itself, and not using text but this is not yet exposed To at least closed source models, so.”

“We think the database will be, like, much thicker than it's been before.”

“There'll be language models running inside the application code, obviously, language models running inside the database as well or large models more broadly, multimodal models will run, you know, everywhere as well.”

“I certainly think probably most enterprises, organizations, companies on earth will have brought language models Into the company, probably in pretty meaningful ways. At minimum, the customer service department, the sales department, ops, back end, legal and h…”