Jul 12, 2023 · 41m · mad
Building LlamaIndex: Jerry Liu on Scaling Retrieval-Augmented AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck interviews Jerry Liu, Co-Founder and CEO of LlamaIndex, about building and scaling the open-source data framework connecting large language models to enterprise data. Jerry details Retrieval Augmented Generation (RAG) architecture, developer design philosophy, enterprise adoption, and his vision for AI-driven knowledge workers.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 16.2% of the talking time here. How this is scored →
Matt as informed peer 3.7 Guest teaching 3.6 Guest disagreement 0.3 Matt pushing back 1.8
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 12:16 Rejecting direct competition with application frameworks
Jerry politely but firmly reframes the question about competition, distinguishing LlamaIndex's deep technical focus on data from broader frameworks like LangChain.
Hardest push from Matt ▶ 16:50 Challenging platform scope vs ETL incumbentsMatt presses Jerry on whether LlamaIndex can realistically span connectors, orchestration, and compute long-term given the massive complexity seen in traditional ETL/ELT startups.
Biggest teaching moment ▶ 6:24 Naive chunking vs production-grade RAGJerry educates the host on the limitations of simple text splitting in RAG architectures, explaining how metadata annotations and relationships are required for production applications.
Matt holds his own ▶ 16:50 Host citing data infrastructure ARR benchmarksMatt demonstrates strong domain expertise by comparing LlamaIndex's potential trajectory to mature $100M+ ARR ELT and orchestration infrastructure companies.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Jerry Liu's Career Journey and LlamaIndex Origin | 3 | 1 | 0 | 0 | Matt opens with detailed background stats on LlamaIndex's GitHub traction and funding round before asking Jerry about his journey. Jerry explains his career trajectory across Quora, Uber ATG, and Robust Intelligence leading into GPT Index. | |
| Overview of LlamaIndex Framework and Core Modules | 2 | 5 | 0 | 2 | Matt asks about the framework components and interjects to ask for a definition of 'right format' during data ingestion. Jerry educates on the RAG pipeline, contrasting naive text chunking with production-grade metadata annotations. | |
| Storage Abstractions and Vector Database Integrations | 3 | 4 | 0 | 1 | Matt expresses surprise at the sheer number of vector databases supported. Jerry details LlamaIndex's storage abstractions, query interfaces, and how the reasoning engine operates over indexed data. | |
| Positioning LlamaIndex in the Generative AI Stack | 5 | 4 | 1 | 3 | Matt demonstrates knowledge of the ecosystem by specifically naming potential competitors like LangChain, Fixie, and Dust. Jerry politely clarifies LlamaIndex's deep specialization in data abstractions compared to general application frameworks. | |
| LlamaHub and LlamaLab Community Projects | 3 | 3 | 0 | 1 | Matt asks Jerry to explain LlamaHub and LlamaLab. Jerry explains LlamaHub's role as a community repository for long-tail data connectors and LlamaLab as an experimental sandbox for agents. | |
| Modern ETL and Data Infrastructure for LLMs | 6 | 5 | 1 | 4 | Matt pushes back by drawing direct comparisons to hundred-million-dollar ETL incumbents and questioning whether one project can own both connectors and compute. Jerry explains how LLM-era ETL differs fundamentally from legacy data pipelines. | |
| LlamaIndex 0.7.0 Release and Modular Architecture | 3 | 4 | 0 | 1 | Matt asks about the recent 0.7.0 release. Jerry details the architectural shift toward lower-level modularity, enabling developers to build bottom-up custom LLM workflows. | |
| Progressive Complexity and Developer Adoption | 5 | 3 | 1 | 3 | Matt challenges Jerry on the tension between catering to beginner simplicity versus power-user depth. Jerry explains the concept of 'progressive disclosure of complexity' adopted from Keras. | |
| Practical Use Cases: Chatbots, OpenBB, and Long-Form Generation | 3 | 4 | 0 | 1 | Matt asks for practical enterprise use cases. Jerry highlights implementations ranging from OpenBB's financial terminal to structured data extraction and long-form document synthesis. | |
| Enterprise Product Vision and Commercial Features | 3 | 4 | 0 | 2 | Matt inquires about enterprise commercialization timelines. Jerry outlines key enterprise capabilities being built, including multi-tenancy, access controls, and production-grade connectors. | |
| Navigating AI Velocity and Rapid Iteration | 4 | 3 | 0 | 2 | Matt asks how Jerry manages product velocity amidst constant AI news. Jerry shares how he balances long-term North Star goals with rapid pivot moments, such as completely rewriting documentation following HackerNews feedback. | |
| Recruiting Strategy and Future AI Vision | 4 | 3 | 0 | 1 | Matt asks about talent acquisition strategies and broad industry outlook. Jerry outlines his vision for automated knowledge workers that reason and execute over data stacks. |
Statements from this episode (16)
LlamaIndex announced an $8.5 million seed round led by Greylock
“You also just announced just last month a 8.5 million dollar seed round. Led by our friends at Greylock.”
Jerry Liu: LlamaIndex aims to connect language models with custom data
“So the high level mission is really to connect your language models with your data and basically unlock the capabilities of language models, whether it's like reasoning agent, like planning, or also like question answering inside extraction on top of your data…”
Jerry Liu: Injecting metadata into text chunks improves LLM retrieval performance
“Second is being able to inject metadata actually is quite important to actually improve retrieval performance of like the downstream application, because like, you know, let's say you're splitting up like a sec, 10 K filing into a bunch of chunks within a sing…”
LlamaIndex is not building a vector database, integrates with 12-20 existing ones
“We're not building our own vector database, but we have a rich set of integrations with, you know like 12 to like 20 different vector databases out there these days.”
LlamaIndex focuses on data infrastructure while LangChain builds broader application frameworks
“Blind train is a great application framework for you to just like get us out of building blocks for a lot of different components, for instance, from like LL modules to prompts to some basic like retrieval and vector database abstractions to like also agent fr…”
Liu: LlamaHub features over 100 community-contributed data loaders
“And so these days, like Lama Hub is a very rich repository of, like, you know, the hundred plus, like, different data loaders from all different services and formats, and it's growing, like, every day.”
Turck: Companies with $100M+ ARR have been built solely on data connectors
“There's like this whole world of ETL or ELT companies on the one hand, and then orchestration companies, and all of those are you know, the entire companies, a hundred million plus ARR companies are being built. Solely on building those connectors.”
Liu: LLM data pipelines differ from traditional ETL stacks due to unstructured content comprehension
“If we're building this new age of L-empowered applications, The kind of requirements for the type of data that like you want to load as well as how you want to extract information from that data will be a little bit different than the existing ETL stack. The r…”
Liu: LlamaIndex enables data querying in three or four lines of code
“Because in about three or four lines of code, you can load data and just it index it and then query it.”
LlamaIndex adopted Keras's design philosophy of progressive disclosure of complexity
“A key philosophy that we launched and like zero dot six auto that we're continuing to build towards is this idea that was inspired by Karis, which is this idea of like progressive disclosure of complexity, where in the beginning things are very high level and …”
Liu: Uber and Instabase use LlamaIndex for enterprise data applications
“We've seen people build these workflows at different settings from, for instance, like hacks on projects at startups building, for instance, like track GPT, like, plugin over, like, your Slack or your Notion all the way to kind of, like, bigger companies, for …”
Liu: OpenBB integrated LlamaIndex as its natural language layer
“OpenBB, which is the open source, like financial analysis tool, and they recently incorporated Lama Index as a natural language layer to help power, like their, basically, open source Bloomberg vibe, right?”
Liu: Generating long-form content over custom data remains a hard problem
“Generating something that's like a paragraph is pretty easy for ChatGPT to do. Generating like an entire blog post or essay, especially over your data is a pretty challenging problem.”
Jerry Liu: LlamaIndex is building a production-grade enterprise version
“So we're Actually building out and scoping out initial version of what, like kind of production grade Lama index would look like.”
Jerry Liu: 'Drop everything' reactiveness accounts for 10-15% of LlamaIndex's work
“There's going to be cases where there's going to be things that come out where like, you know, this really is going to be like a drop everything a moment. Like, you know, just stop what you're doing. Like this actually is like P zero. We have to like figure ou…”