Opinion
Catanzaro: Chinese AI achievements are not driven by a copycat mentality
“I think it's absolutely false to say that you know the achievements of some other country are all being created by sort of, you know, copycat mentality. It's just not true.”
Opinion
Catanzaro: China has been leading in open community-oriented AI development
“I think there's a chance for the rest of the world to catch up to China in the sense that you know, we can understand the benefits of working together as a community to build technologies for AI in a way that I think China has frankly been leading.”
Opinion
Catanzaro: The technological singularity is a wrongheaded idea
“The singularity is, although it's an attractive idea, I think that it's a really a wrongheaded idea because it doesn't really take into account these other factors.”
Opinion
Catanzaro: Open technologies are inherently the safest way to build AI
“I believe that open technologies for AI are inherently the safest way of building AI.”
Assertion Not checkable as stated
Catanzaro: Moore's Law has been economically dead for five to ten years
“The original statement of Moore's law was economic, right? It was about, we can afford to put twice as many transistors on the same chip in every, whatever, 24 months, whatever the time period is. And these days that is, Absolutely not the case. It hasn't been…”
Insight
Catanzaro: At AI compute limits, intelligence gains require higher efficiency
“If you accept as the truth that we're going to be running at the limit, then what that means is that the way to get more intelligence is to be more efficient. We can't get more intelligence by applying more force if we're already at the limit. We have to be mo…”
Insight
Catanzaro: Multi-token prediction lowers inference costs as model accuracy improves
“With multi-token prediction, the speed that you get is a function of the accuracy of your model. The more accurate your model is, the faster the inference is, the cheaper the inference is, the more accurate it is. That's not usually how it works, but in this c…”
Assertion Supported
NVIDIA pre-trained Nemotron Ultra and Super natively using 4-bit floating point
“NemoTron Ultra and Super, ah, were pre-trained using four-bit arithmetic. We pre-trained those in MVFP four”
Assertion Supported
Catanzaro: Nemotron 3's Latent MoE quadruples experts for same inference cost
“Latent MOE is a specific innovation that we have in NemoTron three family. And what it does is actually reduces the amount of communication that has to be sent through NVLink during MOE computations by basically down projecting it. So, you know, every token pr…”
Assertion Not checkable as stated
Catanzaro: MoEs have long been the default architecture in frontier AI
“Yeah, I believe MOEs have been the default in Frontier AI for a long time. They're just a really good combination of inference cost and intelligence.”
Assertion Not checkable as stated
Catanzaro: Enterprise data sovereignty is spurring demand for open AI models
“This is really spurring a lot of demand for open technologies for AI.”
Assertion Supported
Catanzaro: Dario Amodei worked in bioinformatics before deep learning
“At the time he had been working in bioinformatics, so he hadn't been working on deep learning or the things that we call AI these days.”
Assertion Not checkable as stated
NVIDIA DLSS Is About 10 Times More Efficient Than Traditional Rendering
“DLSS is our real-time AI for graphics, and it makes a small GPU run like a big GPU. It's about 10 times more efficient because rather than computing the color of every pixel for every frame, we use AI to infer the color.”
Assertion Supported
NVIDIA DLSS Generates 23 Out of Every 24 Pixels in Games
“These days, 23 out of every 24 pixels is being generated by our AI model when you're using DLSS to play games”
Insight
Catanzaro: Any development or deployment of AI benefits NVIDIA's business
“Whenever AI is further developed and further deployed. It's an opportunity for our business. So, so this is you know, we're very explicitly trying to develop our ecosystem because that's good business for us.”
Assertion Supported
Catanzaro: Combining SSMs and transformers produces smarter AI models than either alone
“Using both of these together was actually better than using either one on their own. And that is independent of the speed benefit. That is just the model is smarter.”
Assertion Partly supported
Catanzaro: Hybrid state-space transformer architectures are widely adopted in frontier AI
“It's become, I think, Quite widely adopted to use some sort of state space model in conjunction with full attention for the base architecture.”
Insight
Catanzaro: Dense models outperform MoE models under strict memory constraints
“You know, they take a lot more memory. If you have a very small amount of memory, a dense model is going to be smarter.”
Disclosure
NVIDIA purchases commercial datasets and opens them when licensing permits
“We do purchase data from companies that that, you know, are building data sets that you can purchase. And to the extent that, you know, we have the rights to redistribute or to open up that data, we do as part of our Mnemotron data effort.”
Disclosure
NVIDIA uses massive compute to generate and publicly release synthetic data
“We also are big believers in synthetic data generation. We use an enormous amount of compute running language models on our own systems to create synthetic data that then helps our models be better at solving problems in specific domains, and we release a lot …”
Disclosure
Catanzaro: NVIDIA's Applied Deep Learning team sits inside GPU division
“My team, for example, is not part of the official NVIDIA research team. My team is actually part of the organization that builds the GPU.”
Insight
Catanzaro: In accelerated computing, software failure destroys hardware value
“Accelerated computing is the composition of thousands of technologies. If any of them fail to deliver acceleration, the value is destroyed. It doesn't matter whether the chip is great if the compiler sucks.”
Assertion Supported
Catanzaro: cuDNN was NVIDIA's first GPU deep learning product
“Then that led to the creation of QDNN, which was NVIDIA's first product for deep learning on the GPU.”
Assertion Supported
NVIDIA details Nemotron 3 parameter specs from 3B to 55B active
“Nano is a thirty billion total three billion active parameter model. Super is one 20 and 12, and Ultra is five 50 and 55.”
