Mar 6, 2025 · 50m · mad
Top AI Researcher on GPT 4.5, DeepSeek and Agentic RAG | Douwe Kiela, CEO, Contextual AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck interviews Douwe Kiela, CEO of Contextual AI and lead author of the seminal 2020 RAG paper, to discuss frontier AI models, the evolution of Retrieval-Augmented Generation, and the transition toward enterprise-grade agentic AI systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 25% of the talking time here. How this is scored →
Matt as informed peer 3.7 Guest teaching 4.6 Guest disagreement 0.8 Matt pushing back 1.4
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 9:40 Debunking the DeepSeek $6M Training Cost Narrative
Douwe forcefully rejects the dominant media framing around DeepSeek's efficiency, arguing that citing a six million dollar single training run ignores the 100x prior R&D computation costs required to discover that run.
Hardest push from Matt ▶ 26:50 Host Defends Content Creators and VCsWhen Douwe criticizes journalists and venture capitalists for creating false binary tech narratives, Matt jokingly pushes back by noting that VCs and content creators increasingly overlap to deliver valuable industry analysis.
Biggest teaching moment ▶ 27:50 The Harry Potter Analogy for Long-Context InefficiencyDouwe educates the host on why long-context model windows cannot replace RAG systems, illustrating how computationally wasteful it is to feed all seven Harry Potter books into a context window just to identify the headmaster.
Matt holds his own ▶ 40:40 Framing Retrieval as an Agentic ToolMatt demonstrates sharp architectural understanding by framing retrieval not merely as a fixed pipeline lookup step, but as a specialized tool dynamically called within multi-step agentic workflows.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Assessing Frontier Model Releases, AI Benchmarks, and Test-Time Compute | 4 | 4 | 2 | 3 | Matt demonstrates solid familiarity with recent frontier model releases, scaling law discussions, and test-time compute. Douwe gently reframes the dichotomy around unsupervised versus reasoning models, explaining that chain-of-thought capabilities mean models like GPT-4o are already reasoning models. | |
| DeepSeek Impact, Training Cost Realities, and Geopolitical Narratives | 4 | 6 | 4 | 2 | Douwe rejects dominant media narratives around DeepSeek, explaining that the reported six million dollar training cost figure misleadingly omits the 100x prior R&D exploration compute. He also re-educates on model hallucination, noting it is ill-defined and often a desirable feature for creative writing tasks. | |
| Origins and Evolution of Retrieval-Augmented Generation (RAG) | 2 | 4 | 0 | 1 | Douwe shares the story of inventing RAG at Meta FAIR using early vector search tools like FAISS. Matt acts as an engaging guide, asking about Meta FAIR's open research culture and early reactions to transformer papers. | |
| Naive RAG vs. Advanced RAG and the Long-Context Debate | 4 | 6 | 2 | 3 | Matt brings up industry debates on fine-tuning and long-context windows replacing RAG. Douwe dismantles these false dichotomies, using a vivid Harry Potter analogy to demonstrate the extreme computational inefficiency of processing massive context windows for basic facts. | |
| Document Extraction and Layout Understanding in RAG Systems | 3 | 4 | 0 | 1 | Douwe details how layout segmentation models process visual structures in enterprise PDFs like tables and charts. Matt asks whether document extraction issues stem from human decisions or technical bottlenecks. | |
| Enterprise Vector Databases vs. On-Demand API Tool Calling | 4 | 4 | 0 | 1 | Matt asks whether future architectures will shift from pre-indexed vector databases to live API tool calling. Douwe explains how on-demand tool calling complements vector search by preserving native enterprise access control. | |
| Contextual Language Models and Grounding to Eliminate Hallucinations | 3 | 4 | 0 | 0 | Douwe outlines Contextual Language Models (CLMs) fine-tuned on top of Llama to strictly ground responses and admit when context is missing. Matt and Douwe highlight Meta's massive contributions to open-source infrastructure like PyTorch and Llama. | |
| Model Alignment and Preference Optimization: From RLHF to KTO and APO | 4 | 5 | 0 | 0 | Matt references specific preference alignment techniques like KTO and APO. Douwe breaks down why moving away from paired RLHF preferences allows enterprises to optimize models from direct feedback on as few as 100 samples. | |
| Building Specialized RAG Agents for Enterprise Applications | 5 | 4 | 0 | 2 | Matt presents a thoughtful mental model of retrieval as a tool within broader agentic workflows. Douwe validates this framing, adding that virtually all high-value enterprise agents are fundamentally specialized RAG agents. | |
| The Role of Synthetic Data in End-to-End RAG Optimization | 4 | 5 | 0 | 1 | Matt asks about practical enterprise adoption barriers and customer expectations regarding AI accuracy. Douwe explains that 100 percent LLM accuracy is impossible, forcing enterprise deployments to manage non-deterministic risks using audit trails and claim verification. |
Statements from this episode (15)
Kiela: DeepSeek proved frontier AI models can rely on synthetic data
“We have kind of an existence proof now that it's actually not that hard to do this and so you don't need to invest all that much in, in data, and you can use synthetic data and get a pretty good model out of that”
Kiela: GPT-4o is already effectively a reasoning model via chain of thought
“I mean, you could argue that GPT-IV-O is also already a reasoning model. It just hasn't been trained on reasoning specifically, but, ah, it can do chain of thought, right? So if it can do chain of thought, it's basically already a reasoning model. It just hasn…”
Kiela: DeepSeek's total development cost was at least 100x its $6M training
“So I would guess that they spent at least a hundred X The amount of that, that single training run, right?”
Kiela: Core language model development is almost solved and plateauing
“It's not even really about language models anymore. That has almost been solved, right? That's kind of why you see things plateauing off a little bit as well.”
Kiela: AI is heading toward specialized language models over generalists
“Where we're headed is that we will have more specialized language models.”
Kiela: FAIR was the first team to build a generative RAG model
“Why RAG became the way you name these things is because it's generative, right? So we were the first ones to have a generative model there.”
Kiela: Attention mechanism, not Transformers, was the real AI breakthrough
“So I would say, and maybe I'm biased because one of my best friends is, is on the original attention paper, but that was the real breakthrough. It's just like figuring out that you have this attention mechanism that actually allows you to yeah, to do a much be…”
Kiela: FAISS was the first vector database
“In the initial paper, we used a vector database or a face. So the words vector database didn't exist at the time. But so face was the first vector database.”
Kiela: Fine-tuning cannot inject new knowledge into AI models
“One common misconception about fine tuning is a lot of people think that you can inject new knowledge into a model using fine tuning. And that is not true.”
Kiela: Long-context LLMs are inherently incredibly wasteful
“Long context models are inherently incredibly wasteful. You're paying for all this compute, and that's maybe why some of the companies that are trying to really sell long context model, long context window models, they will make more money from that, right?”
Kiela: Enterprise RAG fails if complex document data is not properly extracted
“If you want to have a enterprise grade rag system, you are only as good as the data that goes into that rag system. So if you can't extract the data in the right way, so if you have like a sort of table structure and it has like nested information, you can't g…”
Kiela: Advanced RAG systems break down when scaling to a million PDFs
“You can build a very awesome demo on a couple of PDFs and things will probably work. But then you have to scale it up to a million PDFs, and then everything breaks down. And the reason for that is that a lot of these kind of advanced RAG systems still actually…”
Kiela: Aligning language models needs only 100 examples via Anchored Preference Optimization
“So you can train on this when you only have like a hundred examples, you can really make a meaningful, meaningful difference.”