Jun 28, 2023 · 30m · mad
Why Vector Databases Are Exploding: Chroma Co-Founder Jeff Huber on Building AI-Native Infra
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this Data Driven NYC session, FirstMark's Matt Turck interviews Chroma co-founder Jeff Huber on how open-source vector databases provide programmable memory for LLMs, solve hallucinations, and form the backbone of modern AI infrastructure.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 17.3% of the talking time here. How this is scored →
Matt as informed peer 2.9 Guest teaching 3.5 Guest disagreement 0.8 Matt pushing back 0.3
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 28:47 Dismissing traditional database replacement
Jeff forcefully rejects the audience member's premise about vector databases replacing traditional relational databases, calling it 'kind of a dumb question'.
Hardest push from Matt ▶ 18:36 Highlighting hallucination mitigationMatt presses a pivotal structural point that vector databases serve primarily as a grounding layer to limit LLM hallucinations, steering Jeff into detailing grounding prompts.
Biggest teaching moment ▶ 15:22 HTAP architectural lessonJeff educates the host and audience on HTAP (Hybrid Transactional Analytical Processing), detailing why vector databases must handle both continuous transactional updates and bulk analytical operations.
Matt holds his own ▶ 5:43 Explaining vectorization principlesMatt demonstrates strong domain comprehension by independently explaining how unstructured data must be vectorized into semantically meaningful numbers for machine learning consumption.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Welcome and Chroma Funding Background | 3 | 2 | 0 | 0 | Matt opens with background details on Chroma's funding rounds and investors before asking Jeff about his founder journey. Jeff explains his past ML deployment pain points, leading Matt to compliment how established Chroma appears despite its young age. | |
| High-Level Overview of Chroma | 2 | 4 | 0 | 0 | Matt prompts high-level explanations of Chroma and vector embeddings. Jeff takes the lead, using analogies like geographic coordinates on a map and HR vacation policies to educate the audience on semantic search. | |
| Data Vectorization and Embedding Models | 4 | 3 | 0 | 1 | Matt demonstrates technical grounding by noting that unstructured data must be converted into numbers for ML models. Jeff builds on this with a Matrix analogy, and goes on to name key embedding model providers like OpenAI and Cohere. | |
| Developer Experience and Chroma's Roadmap | 3 | 4 | 0 | 0 | Matt asks about Chroma's feature set and roadmap. Jeff details their focus on developer experience, comparing Chroma's trajectory with Lucene/Elasticsearch and outlining open questions around document chunking and nearest neighbor selection. | |
| Open Source Strategy and Cloud Offerings | 3 | 2 | 1 | 1 | Matt inquires about Chroma's Apache 2.0 open source license and monetization model. Jeff plainly asserts that attempt to knee-cap open source features for monetization is dumb, affirming Chroma's commitment to a full open-source standard. | |
| Competitive Landscape and HTAP Architecture | 3 | 5 | 1 | 0 | Matt asks how developers should compare competing vector databases on performance metrics. Jeff educates the audience on HTAP architecture, explaining why vector workloads require both transactional and analytical capabilities. | |
| Emerging Use Cases: 'Chat Your Data' to AI Agents | 4 | 3 | 0 | 1 | Jeff lists emerging use cases like AI agents, comparing early 'Chat Your Data' apps to newspapers on the early internet. Matt interjects with a key industry point regarding vector databases as a tool to limit LLM hallucinations, which Jeff expands upon. | |
| The Emerging Generative AI Infrastructure Stack | 4 | 3 | 0 | 0 | Matt sets up a scenario asking how an enterprise like Moody's should construct a generative AI stack. Jeff maps out software architecture analogies, predicting that the database layer in AI stacks will be significantly thicker than historical databases. | |
| Future Vision for Generative AI | 2 | 4 | 2 | 0 | Matt asks for a 1-2 year prediction in AI, which Jeff lightheartedly rejects as a dead zone, jumping instead to a 3-5 year vision. An audience member asks about failure modes, leading Jeff to explain query density and sparse embedding spaces. | |
| Audience Q&A: Vector DBs vs. Traditional Databases | 1 | 5 | 4 | 0 | Audience member Tony asks if vector databases will replace traditional databases or require SQL-like standards. Jeff pushes back on the premises, stating he dislikes the term vector database and calling the idea of vector DBs replacing relational DBs a dumb question. |
Statements from this episode (16)
Embedding search and analytics enable developers to improve model reliability
“By looking at embeddings doing embedding search, doing analytics over embedding space you could give developers you know, at the minimum of a divining rod, if not a compass, to be able to improve their models and get to the level of reliability they want to ha…”
Huber: Programmable memory enables reliable LLMs across all use cases
“Chroma's belief is that programmable memory, so developers being able to set terministically Hey, language model, this is the knowledge you should know about, this is the knowledge you should use, these are the tools you should know about, these are the tools …”
Chroma is developing an open-source distributed version of its database
“We're working on a distributed version of Chroma. So in the same way that Elastic for those of you that are familiar with TextSearch, Elastic picked up Lucene, made it developer-friendly, they made it distributed, Chroma picks up some of these ANN algorithms, …”
Huber: Chroma is committed to remaining fully open source
“Chroma will always, we are committed to building the ubiquitous open source standard.”
Huber: Vector databases must support both transactional and analytical workloads
“And we think that both certainly transactional has to be the case because it is a online database. It's gonna sit in the loop of applications. Again, you've already seen demos of this happening tonight. But also to make this technology useful for developers, y…”
Huber: 'Chat Your Data' AI Use Case Will See Mass Adoption
“So I think that use case, even just the Chat Your Data use case, truly will go to the ends of the earth.”
Huber: Current SOTA LLMs Lack Reliability for Multi-Agent Workflows
“Now, of course, for those of you that have actually played with technology, I think it's questionable whether the current state of the art Language models, embedding models, et cetera, will give you the reliability you want from, ah, you know, agents working t…”
Huber: Steering LLMs at embedding layer is not exposed in closed-source models
“So, there's lots of stuff around steering language models at the embedding layer itself, and not using text but this is not yet exposed To at least closed source models, so.”
Huber: AI-native databases will be much thicker than traditional databases
“We think the database will be, like, much thicker than it's been before.”
Huber: Multimodal models will run directly inside application code and databases
“There'll be language models running inside the application code, obviously, language models running inside the database as well or large models more broadly, multimodal models will run, you know, everywhere as well.”
Huber: Most enterprises will deploy language models within three years
“I certainly think probably most enterprises, organizations, companies on earth will have brought language models Into the company, probably in pretty meaningful ways. At minimum, the customer service department, the sales department, ops, back end, legal and h…”
Huber: Chroma is building density detection for vector space retrievals
“One of the things that we've been working on is this idea of query relevancy or density. So given retrievals, From vector space, given the search. We can say whether it came from a sparse or dense part of the embedding space.”
Huber: Chroma will not natively manage full human-in-the-loop workflows
“A database specifically, Chroma specifically, is not gonna do all of this workflow, obviously, and we're not gonna have, like, you know, user code for typing in answers.”
Huber: 'Vector database' is too narrow a term for information retrieval
“I don't actually like the term vector database that much. I think it's sort of narrow. I think the job to be done is information retrieval more broadly, and vector search happens to be a useful tool in our toolbox to doing information retrieval.”
Huber: Asking whether vector databases replace classic databases is dumb
“People, there's this, like, you know, big question about, oh, are vector databases gonna replace classic databases? Are these competitive in some way? And I think it's just kind of a dumb question.”
Huber: Vector databases primarily handle un-databased unstructured data
“Vector databases are primarily about unstructured data. It's actually taking data that had no database that knew about it and loading it in for the very first time. Most applications of this stuff is not about taking data that's already in your relational data…”