Jul 2, 2026 · 1h 23m · mad
Inside Nemotron & NVIDIA’s AI Lab | Bryan Catanzaro
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The MAD Podcast, host Matt Turck interviews Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA, covering the architectural innovations behind the open-source Nemotron model family. Catanzaro discusses hardware-software co-design, native 4-bit pre-training, open-source AI philosophy, NVIDIA's unique organizational culture, and the conceptualization of AI as humanity's external brain.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 16.1% of the talking time here. How this is scored →
Matt as informed peer 4.1 Guest teaching 5.0 Guest disagreement 1.2 Matt pushing back 1.7
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 8:21 Rejection of Chinese AI copycat myth
Bryan explicitly rejects the host's framing that Chinese model progress is largely copycat distillation, calling it 'absolutely false' based on his personal experience at Baidu.
Hardest push from Matt ▶ 5:29 Distillation dependence pushbackMatt Turck directly challenges the narrative of open-source momentum by asking whether progress relies heavily on distilling closed models like Anthropic and OpenAI.
Biggest teaching moment ▶ 1:13:30 Singularity reframing via Math Olympiad vs CEOBryan reframes the popular notion of AGI and the Singularity by illustrating that raw intelligence is multifaceted and contextual, contrasting Math Olympiad winners with effective company CEOs.
Matt holds his own ▶ 43:59 Host MoE specialist routing analogyMatt Turck demonstrates sharp domain understanding by constructing an intuitive enterprise analogy for Mixture of Experts (MoE) routing, which Bryan immediately validates.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Open Source Momentum vs. Closed Source Models | 3 | 3 | 1 | 1 | Host Matt Turck opens with an overview of open source momentum and asks Bryan to assess the performance gap between open and closed source models. Bryan provides an agreeable overview, drawing a historical parallel between early closed portals like AOL and the open internet. | |
| Drivers of Open Source Progress & Global AI Dynamics | 4 | 4 | 2 | 4 | Matt pushes Bryan with a provocative question about whether open source relies artificially on distillation from closed models. Bryan gently reframes the question, moving away from competitive scorekeeping to focus on global collective innovation. | |
| Global AI Innovation and Research Contributions from China | 4 | 6 | 3 | 4 | Matt asks whether rapid Chinese AI progress is primarily copycat distillation or genuine novel research. Bryan explicitly rejects the copycat narrative as false, citing his personal career experience at Baidu alongside Andrew Ng and Dario Amodei. | |
| Enterprise Data Sovereignty & Bryan's Career Foundations | 3 | 3 | 1 | 1 | Matt asks for the core enterprise case for open source models over proprietary APIs. Bryan explains data sovereignty, trade secret integration, and custom guardrails as key drivers. | |
| Early GPU Computing, ICML 2008, and Working with Dario Amodei | 4 | 4 | 1 | 1 | Matt invites Bryan to reflect on his career path, noting that pursuing GPU compute in 2008 was a lonely quest. Bryan recounts how academics dismissed his 2008 ICML GPU paper and shares anecdotes about working with a young Dario Amodei. | |
| Rejoining NVIDIA: DLSS, Megatron, and System-Level AI Training | 3 | 5 | 1 | 1 | Bryan describes returning to NVIDIA in 2016 to work on DLSS and starting Megatron in 2017. He explains how Megatron proved to the industry that GPUs could scale large transformer pre-training. | |
| Why NVIDIA Builds Frontier Models & Accelerated Computing Post-Moore's Law | 4 | 6 | 2 | 3 | Matt asks if Moore's Law being dead is official statement. Bryan candidly responds that it has been dead for years economically, forcing NVIDIA to focus on accelerated co-design across hardware and software. | |
| Chronology of Nemotron Models: From Megatron-Turing to Nemotron 3 & 4 | 3 | 5 | 1 | 1 | Bryan walks through the history of Nemotron releases from the 2021 Megatron-Turing 530B model through Llama-Nemotron and Nemotron 3. He humorously acknowledges internal model naming confusions. | |
| The Nemotron Coalition and Ecosystem Pre-Collaboration | 3 | 3 | 1 | 1 | Matt asks about the Nemotron Coalition announced in early 2025. Bryan explains why NVIDIA actively collaborates with ecosystem partners prior to pre-training rather than dropping models unilaterally. | |
| Nemotron 3 Lineup & Pre-Training in 4-bit (NVFP4) | 4 | 6 | 1 | 2 | Matt asks Bryan to clarify 4-bit quantization versus 16-bit precision. Bryan uses a visual posterization analogy to explain NVFP4 and details the mathematical hurdles overcome during 4-bit pre-training. | |
| Hybrid Architecture: Mamba State Space Models and Transformers | 5 | 5 | 1 | 1 | Matt identifies Nemotron's hybrid state-space Mamba and transformer architecture. Bryan details the complementary strengths of constant-memory state-space summaries and lossy-free full attention lookup. | |
| Mixture of Experts (MoE), Blackwell Co-Design, and Latent MoE | 6 | 6 | 1 | 1 | Matt proposes a corporate department analogy for Mixture of Experts (MoE), which Bryan validates and expands using a library analogy. Bryan explains how Blackwell NVL72 dynamic interconnects support latent MoE compression. | |
| Long Context Windows and Multi-Token Prediction (MTP) Speedups | 6 | 7 | 1 | 2 | Matt brings up 1M context windows, context compaction, and multi-token prediction (MTP). Bryan delivers a deep explanation of memory bandwidth bottlenecks, showing how higher predictor accuracy directly accelerates inference speed. | |
| Multi-Teacher Distillation and Nemotron-3 Architecture | 5 | 5 | 1 | 1 | Bryan outlines multi-domain on-policy distillation (MOPD) with 10-15 specialized teacher models. Matt perceptively notes that MOPD solves human organizational friction across 500 researchers as much as technical convergence. | |
| Post-Training Data Sources and Synthetic Data Generation | 5 | 5 | 1 | 2 | Matt asks where post-training data comes from for non-verifiable enterprise domains. Bryan outlines commercial data acquisition and synthetic data generation, re-emphasizing NVIDIA's commitment to releasing data openly. | |
| Reinforcement Learning and Expanding Beyond Verifiable Domains | 5 | 5 | 1 | 1 | Matt asks if reinforcement learning can expand beyond code and math into less verifiable fields. Bryan notes coding's unique status due to execution feedback loops and predicts increasingly complex RL training environments. | |
| NVIDIA's Internal AI Research and Organizational Structure | 4 | 6 | 1 | 1 | Matt explores how NVIDIA structures its internal AI research. Bryan reveals that NVIDIA operates largely outside conventional org charts, drawing volunteer contributions across ten distinct internal divisions. | |
| GPU Resource Allocation and Research Budgeting | 4 | 5 | 1 | 2 | Matt inquires how GPU compute gets allocated internally when research demand is infinite. Bryan describes a bi-weekly hierarchical review process that balances researcher conviction against physical limits. | |
| Balancing Exploratory Research and Bootstrapping Innovation | 4 | 5 | 1 | 2 | Matt asks how NVIDIA balances applied engineering with high-risk exploratory moonshots. Bryan shares his philosophy of bootstrapping research through iterative small-scale validation. | |
| NVIDIA's Company Culture and Leadership Tenure | 4 | 5 | 1 | 1 | Matt observes that NVIDIA maintains an entrepreneurial spirit despite its scale. Bryan attributes this to executive tenure and the cultural maxim 'no one fails alone' in accelerated computing. | |
| Skeptical Perspectives on the Singularity and AI as an External Brain | 4 | 6 | 2 | 2 | Matt references Bryan's known skepticism regarding the Singularity. Bryan explains why he considers the concept wrongheaded, contrasting Math Olympiad talent with CEO capabilities and comparing AI to an external brain. | |
| Addressing Public Perception and AI Backlash | 4 | 4 | 1 | 2 | Matt asks about public AI backlash and whether the tech industry faces a communication challenge. Bryan argues that public anxiety fades as AI becomes invisibly integrated into everyday tools like turn-by-turn navigation. |
Statements from this episode (27)
Catanzaro: Chinese AI achievements are not driven by a copycat mentality
“I think it's absolutely false to say that you know the achievements of some other country are all being created by sort of, you know, copycat mentality. It's just not true.”
Catanzaro: China has been leading in open community-oriented AI development
“I think there's a chance for the rest of the world to catch up to China in the sense that you know, we can understand the benefits of working together as a community to build technologies for AI in a way that I think China has frankly been leading.”
Catanzaro: Enterprise data sovereignty is spurring demand for open AI models
“This is really spurring a lot of demand for open technologies for AI.”
Catanzaro: cuDNN was NVIDIA's first GPU deep learning product
“Then that led to the creation of QDNN, which was NVIDIA's first product for deep learning on the GPU.”
Catanzaro: Dario Amodei worked in bioinformatics before deep learning
“At the time he had been working in bioinformatics, so he hadn't been working on deep learning or the things that we call AI these days.”
NVIDIA DLSS Is About 10 Times More Efficient Than Traditional Rendering
“DLSS is our real-time AI for graphics, and it makes a small GPU run like a big GPU. It's about 10 times more efficient because rather than computing the color of every pixel for every frame, we use AI to infer the color.”
NVIDIA DLSS Generates 23 Out of Every 24 Pixels in Games
“These days, 23 out of every 24 pixels is being generated by our AI model when you're using DLSS to play games”
Catanzaro: Any development or deployment of AI benefits NVIDIA's business
“Whenever AI is further developed and further deployed. It's an opportunity for our business. So, so this is you know, we're very explicitly trying to develop our ecosystem because that's good business for us.”
Catanzaro: Moore's Law has been economically dead for five to ten years
“The original statement of Moore's law was economic, right? It was about, we can afford to put twice as many transistors on the same chip in every, whatever, 24 months, whatever the time period is. And these days that is, Absolutely not the case. It hasn't been…”
Catanzaro: NVIDIA's first major AI model was built jointly with Microsoft
“Our first big model that we trained, we did with Microsoft, right? It was a joint effort where NVIDIA and Microsoft researchers worked side by side to build that.”
NVIDIA details Nemotron 3 parameter specs from 3B to 55B active
“Nano is a thirty billion total three billion active parameter model. Super is one 20 and 12, and Ultra is five 50 and 55.”
NVIDIA pre-trained Nemotron Ultra and Super natively using 4-bit floating point
“NemoTron Ultra and Super, ah, were pre-trained using four-bit arithmetic. We pre-trained those in MVFP four”
Catanzaro: At AI compute limits, intelligence gains require higher efficiency
“If you accept as the truth that we're going to be running at the limit, then what that means is that the way to get more intelligence is to be more efficient. We can't get more intelligence by applying more force if we're already at the limit. We have to be mo…”
Catanzaro: Combining SSMs and transformers produces smarter AI models than either alone
“Using both of these together was actually better than using either one on their own. And that is independent of the speed benefit. That is just the model is smarter.”
Catanzaro: Hybrid state-space transformer architectures are widely adopted in frontier AI
“It's become, I think, Quite widely adopted to use some sort of state space model in conjunction with full attention for the base architecture.”
Catanzaro: Nemotron 3's Latent MoE quadruples experts for same inference cost
“Latent MOE is a specific innovation that we have in NemoTron three family. And what it does is actually reduces the amount of communication that has to be sent through NVLink during MOE computations by basically down projecting it. So, you know, every token pr…”
Catanzaro: MoEs have long been the default architecture in frontier AI
“Yeah, I believe MOEs have been the default in Frontier AI for a long time. They're just a really good combination of inference cost and intelligence.”
Catanzaro: Dense models outperform MoE models under strict memory constraints
“You know, they take a lot more memory. If you have a very small amount of memory, a dense model is going to be smarter.”
Catanzaro: Multi-token prediction lowers inference costs as model accuracy improves
“With multi-token prediction, the speed that you get is a function of the accuracy of your model. The more accurate your model is, the faster the inference is, the cheaper the inference is, the more accurate it is. That's not usually how it works, but in this c…”
Catanzaro: NVIDIA post-trained Nemotron-3 Ultra using multi-domain on-policy distillation
“So with Nemo Tron three ultra, we did post-training using something called multi-domain on policy distillation.”
Catanzaro: NVIDIA used 10 to 15 teacher models for Nemotron-3 distillation
“There's with NemoTron three, I think we had about 10, 10 or 15 of these teachers.”
NVIDIA purchases commercial datasets and opens them when licensing permits
“We do purchase data from companies that that, you know, are building data sets that you can purchase. And to the extent that, you know, we have the rights to redistribute or to open up that data, we do as part of our Mnemotron data effort.”
NVIDIA uses massive compute to generate and publicly release synthetic data
“We also are big believers in synthetic data generation. We use an enormous amount of compute running language models on our own systems to create synthetic data that then helps our models be better at solving problems in specific domains, and we release a lot …”
Catanzaro: NVIDIA's Applied Deep Learning team sits inside GPU division
“My team, for example, is not part of the official NVIDIA research team. My team is actually part of the organization that builds the GPU.”
Catanzaro: In accelerated computing, software failure destroys hardware value
“Accelerated computing is the composition of thousands of technologies. If any of them fail to deliver acceleration, the value is destroyed. It doesn't matter whether the chip is great if the compiler sucks.”