Jun 30, 2023 · 53m · news
Douwe Kiela: Why Data Size Matters More Than Model Size; Why Open Source Isn't Going to Win | E1032 · 20VC with Harry Stebbings
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of 20VC, host Harry Stebbings interviews Douwe Kiela, co-founder and CEO of Contextual AI, who argues that the AI revolution is still in its early stages and explains why data optimization, Retrieval-Augmented Generation (RAG), and Artificial Specialized Intelligence (ASI) represent the future of enterprise AI.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Harry holds 25.1% of the talking time here. How this is scored →
Harry as informed peer 4.6 Guest teaching 5.4 Guest disagreement 3.1 Harry pushing back 3.6
speaking balance: gold is Harry, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:03 Guest dismisses Elon Musk's petition
Douwe forcefully rejects Musk's 6-month pause petition, sarcastically re-framing it as Musk asking everyone else to pause so he can catch up.
Hardest push from Harry ▶ 4:17 Host challenges open source value captureHarry directly interrupts Douwe to reject the premise that Meta's open-sourcing strategy was economically wise, arguing value was given away rather than enshrined.
Biggest teaching moment ▶ 21:54 Guest explains 3 steps to build ChatGPTDouwe clearly and concisely educates the host on the exact technical pipeline required to train instruction-following models: pre-training, supervised fine-tuning, and RLHF.
Harry holds his own ▶ 11:33 Host counters with Emad Mostaque quoteHarry brings deep domain preparation by pitting Stability AI founder Emad Mostaque's claim that hallucinations are a feature against Douwe's claim that they are an enterprise dealbreaker.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Harry as informed peer | Guest teaching | Guest disagreement | Harry pushing back | Why |
|---|---|---|---|---|---|---|
| Facebook AI Research (FAIR) and the Impact of Open Source | 5 | 3 | 2 | 5 | Host interrupts to challenge the guest on value capture, questioning whether Meta lost value by over-indexing on open source. Guest politely reframes around Meta's broader corporate mission beyond purely maximizing revenue. | |
| Experience at Hugging Face and Community Building | 3 | 1 | 1 | 1 | Host asks about Hugging Face's company culture and shares an anecdote about their custom swag policies before transitioning to the founding of Contextual. | |
| LLMs Enterprise Readiness and RAG Architecture | 2 | 6 | 1 | 2 | Guest details why LLMs fail enterprise requirements and explains how Retrieval Augmented Generation (RAG) decouples memory from generation to solve hallucination and privacy. | |
| Model Transparency and RAG's Attribution Power | 6 | 6 | 3 | 5 | Host probes model transparency and challenges guest with a counter-quote from Emad Mostaque claiming hallucinations are a feature, not a bug. Guest reframes the quote with nuance. | |
| Model Agnosticism in a Rapidly Evolving Field | 5 | 5 | 2 | 4 | Host presents a guest thesis that model-agnostic startups will win. Guest agrees short-term but explains Contextual's strategic choice to focus on Specialized Intelligence (ASI) over AGI. | |
| AGI vs. Artificial Specialized Intelligence (ASI) | 3 | 7 | 2 | 3 | Guest educates the host on model scaling trade-offs, citing Sam Altman and Meta's Llama paper to show data volume matters more than parameter count for optimal performance. | |
| The Role of Proprietary and Pre-trained Data | 6 | 7 | 2 | 4 | Host cites VC rationale for rejecting AI wrappers lacking proprietary data. Guest breaks down the technical steps to build ChatGPT (Pre-training, SFT, RLHF) and explains data distillation using GPT-4. | |
| Moat Debate: OpenAI vs. Open Source | 5 | 7 | 7 | 5 | Host raises the famous leaked Google memo claiming OpenAI and Google have no moats. Guest strongly rejects the memo's author as clueless and labels open source parity expectations naive. | |
| Model Security and Next-Generation Cybersecurity | 5 | 6 | 2 | 3 | Host asks whether AI security tools will be built natively by model providers or third-party security vendors. Guest categorizes the AI landscape using a three-tier pyramid model. | |
| Existential Risk, Self-Interest, and the AI Pause Petition | 4 | 6 | 8 | 4 | Guest ridicules Elon Musk's AI pause petition as self-serving and calls x-risk narratives incumbent fear-mongering designed to secure regulatory capture. | |
| AI Regulation and the Chasm of Knowledge | 5 | 5 | 5 | 3 | Host highlights proposed EU rules threatening model usage. Guest criticizes the tech literacy gap in regulatory bodies, citing infamous Congressional hearings, and warns EU over-regulation will destroy innovation. | |
| Enterprise AI Adoption and Data Sovereignty | 6 | 5 | 3 | 4 | Host cites a major European bank refusing to send data off-premise. Guest explains how Contextual's hybrid architecture keeps data inside the customer's virtual private cloud. | |
| Decoupling the Data Plane and Model Plane | 5 | 6 | 3 | 5 | Host pushes guest on whether VC valuations for AI startups have become absurd. Guest defends high valuations based on massive potential market payoffs. | |
| Winners and Losers of the Tech Giants | 4 | 5 | 3 | 2 | Host asks for a breakdown of winner and loser tech giants. Guest lauds Microsoft's Open AI pivot while noting Apple's lack of progress in AI. |
Statements from this episode (38)
Douwe Kiela: LLMs generate inaccurate statements with high confidence
“These models make things up with very high confidence.”
Douwe Kiela: Inability to delete LLM data creates GDPR compliance issues
“So we can't really remove information from them, which is kind of tricky from a GDPR perspective.”
Douwe Kiela: Applied AI research creates more value than theoretical tangents
“Having a very clear Real world application for the research that you're doing makes it much more valuable than going off on a tangent and maybe being a bit too far ahead of the rest of the field.”
Douwe Kiela: Current AI breakthroughs would not happen without Meta's PyTorch
“So PyTorch really without PyTorch, none of this stuff would be happening right now. And so it's really like fundamental for all of the AI breakthroughs.”
Douwe Kiela: Hugging Face excels at marketing and community building
“What really impressed me actually is how good they are just at marketing and branding and community building.”
Douwe Kiela: Generative AI was not ready for enterprise adoption post-ChatGPT
“We saw this kind of great excitement in the world, but at the same time, a lot of disappointment about it not being quite ready yet for real world adaption in, in enterprises where you actually want to use this technology.”
Douwe Kiela: The generative AI market is far from settled
“We think it's still very early innings in the game, so I think a lot of people sometimes think that it's you know, the game has been played, but it's just getting started.”
Douwe Kiela: I co-created Retrieval-Augmented Generation (RAG) at Facebook AI
“We're specifically basing it on this retrieval augmented generation, which is something that me and my colleagues at fair came up with in.”
Douwe Kiela: Decoupling LLM memory reduces hallucinations and enables data deletion
“You decouple the memory from the generative capacity of the large language model, and this allows you to ground the generations from the language model into things you retrieved. In your memory, essentially. So you get much less hallucination. You get attribut…”
Douwe Kiela: Humans will never fully understand large neural network outputs
“So we're not going to be able to really know why a neural net, what does what it does at the scale that neural networks operate at.”
Douwe Kiela: AI hallucinations aid creativity but destroy enterprise value
“So I think in some cases it is a feature. If you want to use a language model for creative writing and if you want it to be really, really creative, then you probably want it to hallucinate. So in a way it's a spectrum of groundedness and hallucination where i…”
Douwe Kiela: Model-agnostic AI startups will have a competitive advantage
“At this particular point in time, probably yes, but just because the field is moving so incredibly quickly. And so I think that in the next year, we're going to see lots of other models coming out. And if you can have a language model, agnostic AI company that…”
Douwe Kiela: Anthropic and OpenAI are consumer-facing companies chasing AGI
“So if you look at anthropic and open AI, I think they're really chasing for this idea of AGI and they're relatively consumer facing.”
Douwe Kiela: The AI industry lacks a clear definition of AGI
“It's slightly better defined now, but one of the big issues has always been that we don't really know what that term even means.”
Douwe Kiela: Artificial Specialized Intelligence will be achieved much faster than AGI
“And I think you're totally right that that's a much easier problem to solve much quicker and then slowly grow with the capabilities of these models.”
Douwe Kiela: Data size matters more than model size for AI performance
“It's just that data size matters even more than model size. And I think the Lama paper out of Meta really brilliantly showed this. Where if you train a smaller model on more data for longer, then you get a better model. So you get more bang for your buck if yo…”
Douwe Kiela: Meta's LLaMA model used zero proprietary training data
“So the Lama model was not trained on any proprietary data. It was just trained on open data on the web”
Douwe Kiela: GPT-4 will disrupt Mechanical Turk before knowledge workers
“And so, so GPT-IV might end up disrupting, not like knowledge workers necessarily, but it might just disrupt like mechanical Turk and is just a, an annotator on steroids.”
Douwe Kiela: OpenAI built a massive, underutilized data moat from ChatGPT
“And they haven't even really trained as far as I know on the data that comes out of ChatGPT going viral, right? So they had ChatGPT, it went viral. This led to this giant, giant data mode that they haven't even really used, used yet.”
Douwe Kiela: The leaked 'no AI moat' Google memo is completely wrong
“I think in terms of data modes and maybe you, you've seen this come by actually, there was this Google memo from an internal Google employee who had written that open AI and Google have no mode. I think for me as a AI researcher, when I read that memo, I was l…”
Douwe Kiela: Huge market opportunity exists for an AI evaluation rating agency
“I think there's a giant opportunity in the market actually for. A startup or several startups becoming like the Moody's or the SMP sort of you know the folks who evaluate the quality of AI for specific use cases because nobody really knows.”
Douwe Kiela: GPT-4's coding skills may be inflated by dataset contamination
“Data contamination where a bunch of these language models are trained on the things that they are being evaluated on. So GPT-IV looks like it's an amazing coder, but it might also just be trained on the data that it's evaluated on, which means that it's not ac…”
Douwe Kiela: AI model protection will drive a new cybersecurity wave
“Oh, absolutely. Yeah. Yeah. So that's completely going to change everything.”
Douwe Kiela: Foundation model builders will rely on external AI security audits
“And so that's a, an interesting part of the market, but I don't think that, that the actual foundation model builders like open AI and contextual are going to build that technology in house. It's probably gonna be an external sort of audit.”
Douwe Kiela: The AI market will be tiered across multiple specialized models
“It's going to be lots of models at different parts different layers of this pyramid being used for different kinds of applications.”
Douwe Kiela: The open-source AI ecosystem exists solely due to Meta's LLaMA
“This whole flourishing that you see right now of open source models that basically comes from Meta's generosity in giving Lama away for free. And if they hadn't done that, then you wouldn't see that.”
Douwe Kiela: Getting struck by lightning is more likely than AI extinction
“Probably the chance of me getting hit by lightning, like right now is much higher than that happening.”
Douwe Kiela: Incumbent AI firms promote existential risk to entrench regulatory advantage
“So the people who are pushing this narrative are really the people who are benefiting from this being the narrative. So these are the incumbent AI companies who are, who want to have either the market regulated. Right. Because then they benefit because they ca…”
Douwe Kiela: EU AI regulation will completely destroy innovation
“What Europe is going to try to do is overregulate everything and just completely destroy
innovation.”
Douwe Kiela: An enterprise AI adoption tidal wave is coming
“I think the tidal wave is coming. There are just big problems that we have to overcome, and these are the things I just talked about, right? So hallucination, attribution compliance up-to-dateness, data privacy, latency and I think the whole field is, is movin…”
Douwe Kiela: Large AI startup funding rounds are justified by potential payoffs
“I think some of the rounds were pretty big, but I think it's also justified just because this stuff is really going to change the world. And so one bet, if it's right, has massive payoff.”
Douwe Kiela: AI hype disillusionment will dry up funding and threaten startups
“At some point there's going to be a disillusionment with the technology and then funding might dry up and then these places are really in trouble.”
Douwe Kiela: Microsoft repositioned itself as an AI leader via OpenAI
“So I, I've been very impressed actually by how Microsoft has managed to turn everything around by strategically collaborating with a better AI lab in the shape of open AI. And they've just really turned that into this narrative where Microsoft is an AI leader …”
Douwe Kiela: Apple has failed to produce interesting AI innovations
“So far I haven't really seen a lot of interesting things coming out of Apple.”
Douwe Kiela: AutoGPT does not actually work despite the hype
“And there's an auto GPT thing that is going to change the world, but doesn't actually work.”
Douwe Kiela: I was wrong to underestimate compute scaling laws and OpenAI
“The strongest belief I had that turned out to be wrong is that I really underestimated how important scale is in artificial intelligence. So I, and I think this is really one of the things that open AI has excelled at is that if you throw an order of magnitude…”
Douwe Kiela: AI will displace the majority human workforce within 5-10 years
“If you look at how OpenAI and Anthropic and these places define AGI, it's as systems achieving capabilities that allow them to effectively do the work of humans for the majority of economically valuable human tasks. Then we're not that far away from it. And so…”
Douwe Kiela: OpenAI and Anthropic are the Lycos and AltaVista of AI
“So if all the stars align, then OpenAI and Anthropic and all of these places, they had this great first generation technology. So they're kind of like the Lycos and Alta Vista of search engines. And the technology we have is more like PageRank. And that would …”