Dec 15, 2025 · 40m · big-technology
Capital One's Prem Natarajan: Why We're Building Our AI From The Ground Up
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Capital One Chief Scientist Prem Natarajan explains why the financial institution builds its AI infrastructure and customizes open-source models from the ground up, highlighting real-world deployments of Agentic AI, full-stack risk governance, and multidisciplinary operational readiness.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Alex holds 17.1% of the talking time here. How this is scored →
Alex as informed peer 4.3 Guest teaching 5.6 Guest disagreement 1.0 Alex pushing back 3.1
speaking balance: gold is Alex, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:24 Rejecting high AI pilot failure rates
Prem directly dismisses Alex's cited study claiming 95% of enterprise AI pilots fail, stating it is completely divorced from Capital One's operational reality.
Hardest push from Alex ▶ 23:15 Challenging open-source necessity using Anthropic's claimsAlex brings up specific counterarguments from Anthropic CEO Dario Amodei, questioning why Capital One insists on open-source ground-up builds when closed frontier models offer deep data integration.
Biggest teaching moment ▶ 17:25 DARPA speech translation systems lessonPrem uses his DARPA background in speech recognition and machine translation to teach Alex why loose modular integration fails without synchronized, end-to-end full-stack architectures.
Alex holds their own ▶ 28:40 Drilling into the post-DeepSeek open-source raceAlex demonstrates industry depth by citing the post-DeepSeek market shifts and community dynamics, challenging Prem on whether open-source models have genuinely achieved parity with closed frontier labs.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Alex as informed peer | Guest teaching | Guest disagreement | Alex pushing back | Why |
|---|---|---|---|---|---|---|
| Capital One's Architecture and Risk Tech Foundation | 4 | 5 | 1 | 4 | Alex presses Prem to clarify whether Capital One is genuinely building from scratch or merely fine-tuning off-the-shelf models with internal data. Prem explains their cloud platform layers, GPU infrastructure, and risk-management tooling. | |
| Defining Agentic AI and Launching Chat Concierge | 3 | 6 | 0 | 2 | Alex asks for a practical breakdown of Capital One's generative AI stack. Prem defines agentic AI as the synthesis of reasoning and specialization, detailing the deployment of Chat Concierge in auto financing. | |
| How Chat Concierge Transforms the Auto Buying Experience | 4 | 5 | 1 | 2 | Alex posits a hypothetical consumer walking into a dealership, but Prem corrects the premise by pointing out that car buying journeys begin online with dealer discovery. | |
| Capital One's Business Motivation and Customer-Centric Focus | 5 | 6 | 1 | 5 | Alex asks why a retail bank is building software for car dealerships and presses on where proprietary data creates an advantage. Prem draws on his DARPA speech-recognition experience to explain why integrated full-stack builds outperform fragmented systems integration. | |
| Expanding Agentic AI to Agent Assist and Developer Workflows | 3 | 5 | 0 | 1 | Alex asks what follows the initial dealership beachhead. Prem details internal call-center Agent Assist to reduce cognitive load on human representatives, as well as developer productivity tools. | |
| Why Customize Open Source Models over Closed APIs | 6 | 6 | 1 | 6 | Alex cites conversations with Anthropic's Dario Amodei regarding closed models supporting customization, pushing Prem on why open source is necessary. Prem explains how model distillation, low latency, and deep parameter control necessitate open-source customization. | |
| The Enterprise AI Race and Baking Context into Models | 6 | 6 | 2 | 3 | Alex references DeepSeek and asks whether open source is beating closed models in the AI race. Prem reframes the race from public benchmarks to enterprise value generated by baking persistent domain context into models. | |
| Overcoming AI Pilot Failure Through Multidisciplinary Readiness | 5 | 5 | 3 | 4 | Alex cites studies claiming 95% of enterprise AI pilots fail. Prem directly rejects that high failure rate applies to Capital One, explaining that multidisciplinary talent and strict vetting processes ensure production viability. | |
| Historical Evolution of AI from DARPA and Alexa to Banking | 3 | 6 | 0 | 1 | Alex asks Prem to reflect on the evolution from early Amazon Alexa development to modern banking AI. Prem emphasizes the foundational contributions of DARPA before highlighting Alexa's milestone as mass-market ambient consumer AI. |
Statements from this episode (15)
Capital One operates entirely on public cloud infrastructure
“We're the first company first bank that went all in on the cloud. We're entirely on the public cloud.”
Enterprises must deeply customize AI models on proprietary data
“In order for you to truly bring all of that to life for your customers, you have to bring your own data to the models. In a way that you can do deep customization of those models so that you truly unlock the value in that data for the products and services tha…”
Cloud GPUs lack elasticity and availability remains a persistent challenge
“GPUs, as you know, are still not elastic. In fact, they're often, you know, availability itself is, can be a challenge.”
Agentic AI combines reasoning with model specialization for complex workflows
“Agentic AI, at least the way we look at it, is the is the bringing together of two of the most powerful forces in generative AI today. One is the power of reasoning, and the other is the power of specialization, and so agentic AI uses reasoning to break comple…”
Capital One's Chat Concierge AI is live at nationwide auto dealerships
“Chat Concierge is now available. At many dealerships across the company across the country.”
Reframing generative AI as agentic workflows bridges lab-to-production gaps
“Casting what seems like generative AI problems into agentic problems has allowed us to bridge the gap between the lab and production.”
The most satisfying customer service experiences retain a human in the loop
“The most satisfying experience always have a human in the loop at some point.”
The vast majority of AI performance gains occur post-production
“Building something in the, ah, in your engineering environment and then taking it to production is one step of it. Ah, what I've come to recognize is that's the first step in the AI stairway to heaven. Right? A lot of the action, a lot of the learning is actua…”
Off-the-shelf AI fails because complex architectures require full-stack joint optimization
“We are in, in my mind, Past a system integration view of the world where you simply say, I bring this in here, I bring this in here, I tie them together and do it. I'll give you an example that's very old, ah, from speech recognition and machine translation, l…”
AI's primary objective is offloading human cognitive burden during peak strain
“I've always felt one of the noble aims of AI, Alex, is to transfer a cognitive burden from the human to the system at a time when the human feels that burden to be most heavy.”
Specialized, compact AI models match performance while significantly reducing latency
“One of the other benefits of specialization is the models can be made much more compact to deliver the same kind of performance relative to the size of the model, right? Because now they're specialized, which means they can run much faster, which means overall…”
Closed-source models will continue driving value in broad horizontal enterprise tasks
“There will be areas, for example, like software development, right, where there is so much horizontal aspect to it in terms of how it is practiced across the world. Where I do think these other approaches, like, you know, where what you bring to that is the co…”
Capital One uses multiple open-source AI models but favors US-based providers
“We actually use more than one depending on the application. Again, like I said, we have this benchmarking, so we take a look at these models and I mean, we favor US-based models in, in general but beyond that, we're constantly looking for which model provides …”
Baking static context directly into models outperforms relying on dynamic prompting
“One of the best ways to bring the right context to the models is actually to do a deep customization of the models. Not all context is dynamic. A lot of context is also, you know, changes at a much slower rate. That context being baked into the model makes the…”