Dec 20, 2023 · 44m · mad
The Race to Build the Ultimate AI Programmer | Poolside CTO Eiso Kant
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In Poolside's first public interview on The MAD Podcast, CTO and Co-Founder Eiso Kant discusses the company's $126 million seed round and its mission to build foundational AI models for software development. Kant details Poolside's technical innovations in Reinforcement Learning from Code Execution Feedback (RLCF), custom training architecture, and commercial roadmap.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 8% of the talking time here. How this is scored →
Matt as informed peer 3.0 Guest teaching 4.9 Guest disagreement 1.3 Matt pushing back 1.1
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 31:30 Eiso forcefully dismissing skeptics of synthetic data
Eiso strongly challenges prevailing industry skepticism regarding synthetic training data, arguing that verifiable code output indisputably proves models can learn from generated data.
Hardest push from Matt ▶ 8:49 Matt challenging guest to move past existential philosophy to product focusMatt politely but directly interrupts a broad existential thesis to demand how Poolside concretely narrows its focus to build a real product.
Biggest teaching moment ▶ 14:37 Eiso breaking down the distinction between RLHF and RLCFEiso educates the host on how standard pre-training resembles reading textbooks whereas RLCF provides deterministic feedback analogous to doing end-of-chapter exercises.
Matt holds his own ▶ 14:08 Matt quoting precise technical definitions from company materialsMatt demonstrates high preparation by accurately quoting Poolside's technical literature on reinforcement learning from code execution feedback across tens of thousands of repositories.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Founder Backgrounds and the Origins of Poolside | 3 | 3 | 0 | 0 | Matt demonstrates clear background knowledge about the guest's co-founder Jason, filling in details about his VP role at Heroku and stint as a Redpoint VC. Eiso shares early history about founding Sourced and meeting Jason during GitHub's acquisition attempt. The exchange is warm and highly collaborative. | |
| The Vision Behind Poolside and Software Development as an AGI Proxy | 1 | 5 | 1 | 0 | Matt asks a single broad question about Poolside's core vision, allowing Eiso to deliver an extended monologue on AI existentialism and framing software development as an AGI proxy. Eiso educates the audience on world models, planning, and reasoning as fundamental pillars of intelligence. | |
| Product Roadmap and Market Sequencing | 2 | 4 | 1 | 2 | Matt prompts Eiso to ground his broad AGI vision into an actual product strategy. Eiso outlines his sequencing framework and predicts widespread adoption of developer AI assistants within 24 months. | |
| Model Capabilities and Directly Challenging GitHub Copilot | 3 | 5 | 2 | 2 | Matt presses on whether Poolside is starting with specific languages or stack layers. Eiso gently reframes the question away from specific languages toward the fundamental capabilities race versus go-to-market race, revealing their plan to compete directly with GitHub Copilot X. | |
| Reinforcement Learning from Code Execution Feedback (RLCF) | 5 | 6 | 1 | 2 | Matt cites specific language from Poolside's documentation regarding reinforcement learning from code execution feedback (RLCF). Eiso provides a deep technical breakdown comparing pre-training to reading textbooks and RLCF to working through exercise sets with execution feedback. Matt follows up with a probing question on non-human code solutions. | |
| Data Quality and Synthetic Data Generation | 3 | 6 | 2 | 1 | Matt directs the topic to synthetic data generation and data curation. Eiso presents a comprehensive explanation of token quality and rejects the common criticism that AI models cannot learn from synthetic data, drawing analogies to academic textbooks. | |
| Engineering Culture, Custom Infrastructure, and Reversible Networks | 4 | 6 | 2 | 1 | Matt makes an insightful point regarding the heavy role of systems engineering relative to pure ML in frontier AI labs. Eiso explains Poolside's choice to build custom training infrastructure from scratch and train large models using reversible networks despite industry skepticism. | |
| Strategic Talent Acquisition and Building in Europe | 3 | 4 | 1 | 1 | Matt brings up Poolside's notable decision to build its primary engineering footprint in Europe. Eiso explains their candidate mapping data and strategic decision to 'zig while others zag' rather than competing directly in San Francisco. |
Statements from this episode (17)
Kant: Sourced had Copilot-style code generation working in 2017
“And we had a lot of the things running things that today that look like copilot where you could, you know, get code suggestions for based on natural language instructions.”
Kant: Neural networks will learn human-level capabilities in our lifetime
“That it's very like that in our lifetime, neural networks will become capable of learning anything and everything that we are capable of as humans.”
Kant: Software development is the best proxy task for general AI intelligence
“Software development in our opinion, being a pretty good proxy task. For a big part of the spectrum of intelligence. You need an understanding, a model of the world, because we built software for the world. You need to be particularly strong at reasoning and y…”
Kant: Executable source code allows automated AI model feedback
“Source code is one of the very few things that we generate with neural networks. That we can actually execute an introspect. It doesn't require human feedback to evaluate it. It doesn't require humans to step by step reason through it. We have built compilers,…”
Kant: Most developers will adopt AI pair programmers within two years
“Well, the first thing that I think is at this point, almost uncontested, but frankly, when I was building source, most people didn't think would happen is the fact that the majority, if not all developers in the next couple of years in the world will adopt som…”
Kant: Non-engineers will build software via AI prompts within three years
“We don't think we're more than three years out to that point where a non-engineer, someone with no coding, you know, software development background is able to come and give one of those high level vague things of what they want built, right? Cause that's what…”
Kant: Poolside is vertically integrating custom foundation models and offering APIs
“We deliver this AI pair programming assistant full application stack with the models, I think fully vertically integrated with our own foundation models trained from scratch, but then also exposing those models with an API to allow other people to build all th…”
Kant: Model capabilities determine 80-90% of AI product value
“The capabilities of these products are determined today by the capabilities of the models underneath, right? About 80, 90%.”
Developers abandon tools quickly when products with superior AI emerge
“They're the first people to leave you if something else is more capable.”
Poolside will launch a direct competitor to GitHub Copilot X in 2024
“But in terms of what we're delivering to the end user, we're starting with a direct competitor to get up Copilot X. And so natural language, multi-turn dialogue system. Where developers give instructions of what they want done while they're in their editor. Th…”
Eiso Kant: Poolside pre-trains models on a 50/50 language-code split
“Our data split is about 50, 50 between language and code.”
Kant: Programmatic RL can scale magnitudes larger than human feedback
“And it's that RL loop that is very interesting because since it's programmatic, since we have an Oracle of truth, we can scale this up far larger, right? Magnitudes larger than what you can do with human feedback today.”
Eiso Kant: AI will recycle world data into synthetic data within years
“And so this is kind of our view and our, if you extend this a couple of years out, and this is where some people will probably have objection with us. We think that this will go so far. That we will kind of recycle all of the world's real data into higher qual…”
Poolside built custom distributed LLM training framework from scratch
“We didn't take Megatron or DeepSpeed or kind of the big open source frameworks for training large language models. As our base, we built our own from scratch and we built our own distributed training for our pre-training stage.”
Kant: Poolside is training world's largest models using reversible layers
“We've decided to scale up rev nets, reversible layers. So, so to the best of our knowledge, I think we're training the world's largest models using reversible layers that, that we've seen at least that nothing else that we're aware of publicly.”
Poolside found top AI talent split 50/50 between North America and Europe
“We went through about 3200 or 3200 people that we ended up putting on the list. And we had this location column, like think genuinely Google spreadsheet, you know, like where's someone based. And at some point we looked at all these locations and we're like, w…”
Poolside plans enterprise strategy targeting companies with up to 100k developers
“And so you're going to see a strong emphasis of us of also bringing poolside into enterprises, into complex environments where there might be 10,000 or a 100,000 developers, where the model needs to continuously learn and be fine-tuned on their data, where it'…”