Mar 6, 2024 · 1h 37m · latent-space
Open Source AI is AI we can Trust — with Soumith Chintala of Meta AI
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of the Latent Space Podcast, PyTorch creator and Meta AI Fellow Soumith Chintala explores the architectural evolution of deep learning frameworks, the strategic imperative for open-source AI, and emerging research frontiers spanning custom silicon, household robotics, and digital olfaction.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 5.1 Guest teaching 6.4 Guest disagreement 2.8 The hosts pushing back 2.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 33:34 Pushing back on the synthetic data hype cycle
Soumith bluntly rejects the prevalent industry claim that synthetic data is a revolutionary magic wand, explaining that it is only effective where low-rank symbolic world models already exist.
Hardest push from the hosts ▶ 45:47 Challenging compute allocation on steep loss curvesAlessio presses Soumith on why Meta stopped training Llama 2 70B when its loss curves remained steep, directly asking if training was prematurely cut due to infrastructure constraints.
Biggest teaching moment ▶ 8:55 Technical breakdown of PyTorch operator explosionSoumith explains the unavoidable physical constraints of GPU/CPU memory hierarchies and compilation times, showing why generic AI frameworks cannot reduce down to minimal operators like TinyGrad without unacceptable performance trade-offs.
The host holds their own ▶ 4:25 Alessio frames PyTorch vs TinyGrad architectural tensionAlessio demonstrates technical depth by referencing George Hotz's CISC vs RISC framing and contrasting PyTorch's 250+ primitive operators against TinyGrad's minimalist core.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| PyTorch Architecture, Operator Complexity, and Comparing with TinyGrad | 6 | 7 | 4 | 3 | Alessio sets up the discussion by comparing PyTorch's architectural complexity with George Hotz's TinyGrad approach. Soumith delivers a comprehensive technical breakdown explaining why PyTorch requires vast operator complexity due to hardware memory hierarchies and input tensor shapes, rejecting the notion that TinyGrad's minimal operator count can scale generally without severe compile-time trade-offs. | |
| Framework Neutrality, Hardware Ecosystems, Mojo, and Apple MLX | 6 | 6 | 2 | 3 | Alessio and Swix probe into hardware neutrality, Mojo interoperability, and Apple MLX. Soumith details PyTorch's demand-driven philosophy, clarifying that Mojo cannot easily augment PyTorch's front-end and explaining how MLX will inevitably run into the exact same distributed scaling complexities if it moves beyond Mac hardware. | |
| AI Framework History, Inference Services, and Benchmark Integrity | 5 | 6 | 3 | 2 | The hosts inquire about FAIR alumni startups and recent benchmark controversies involving AnyScale. Soumith analyzes the economics of LLM inference as a low-margin 'laundromat' model where bespoke kernel optimization moats rapidly evaporate within months due to narrow architectural problem spaces. | |
| Exotic PyTorch Applications, Neuro-Symbolic AI, and Synthetic Data Realities | 5 | 8 | 5 | 3 | When the hosts bring up synthetic data hype, Soumith firmly debunks the popular narrative that synthetic data is a magic wand. He systematically educates them on how synthetic data only works when grounded in human-derived symbolic models, as neural networks lack intrinsic mechanisms to ingest low-rank symbolic world models directly. | |
| Evolution of Meta AI Models: OPT, Llama Series, and Compute Allocation | 6 | 6 | 2 | 3 | Alessio asks technical questions regarding training loss curves and GPU capacity allocation across Meta's infrastructure. Soumith clarifies the history between OPT, Llama 1, and Llama 2, explaining that GPU allocation decisions are standard operational trade-offs governed by time constraints and data readiness rather than arbitrary compute ceilings. | |
| Research Strategy, Career Guidance, and Meta's Custom Silicon (MTIA) | 5 | 6 | 2 | 2 | Alessio asks about avoiding research mode collapse and career strategies for PhDs, followed by Swix asking about Meta's MTIA custom silicon. Soumith delivers advice on balancing fundability with intrinsic motivation and explains the economic and power efficiency math behind specialized datacenter ASICs. | |
| The Open Source AI Philosophy, Corporate Incentives, and Global Trust | 5 | 7 | 4 | 2 | Soumith delivers an extended, passionate exposition on the philosophy of open source AI, contrasting corporate safety rhetoric with decentralized accessibility. He challenges closed-source alignment views by arguing that trust in centralized AI correlates with whether an individual grew up trusting or distrusting their government. | |
| Overcoming Open Source Coordination Issues with a Unified Feedback Sinkhole | 4 | 7 | 3 | 1 | Soumith diagnoses open source AI's fatal weakness: a lack of coordinated human feedback sinks compared to OpenAI and Google. He lays out a concrete system proposal for open front-ends to route user feedback into a centralized, filtered repository to overcome closed-lab data flywheels. | |
| The Continuous Path to AGI and Decentralized Evaluation Benchmarks | 5 | 6 | 3 | 2 | Alessio queries whether feedback loops push towards personal utility or true AGI. Soumith deconstructs economic definitions of AGI, arguing progress is a continuous evolutionary continuum, and points out the inherent sampling biases of centralized leaderboards like LMSYS Arena. | |
| Beyond Text: Home Robotics Research at NYU, UX, and Hardware Limits | 5 | 6 | 2 | 2 | The hosts transition to robotics and physical AI applications. Soumith discusses his NYU research, emphasizing that sample efficiency and hardware mechanical reliability are vastly greater bottlenecks than pure deep learning model architectures. | |
| Digital Olfaction with Osmo AI and Concluding Reflections | 4 | 5 | 1 | 1 | Swix introduces Osmo AI and digital olfaction. Soumith illustrates how primitive digital smell is compared to vision and sound, outlining the path from near-term scent synthesis to ubiquitous sensory integration before closing on intrinsic motivation. |
Statements from this episode (29)
Chintala: PyTorch is around 190,000 lines of code
“PyTorch is like a 190,000 lines of code or something at this point.”
Chintala: Simple AI frameworks must accept long compile times
“You can write a very simple framework but then you also should be willing to eat the long compile times of, like, searching for that optimal performance at runtime.”
Chintala: George Hotz's TinyGrad requires major breakthroughs to match PyTorch
“There's no, like, I don't think, like, unless we have, like, great breakthroughs, like, George's vision is achievable, like, or, like, he should be thinking about a narrower problem, such as, I'm only gonna make this for, like, work for self-driving car con ne…”
Chintala: Deep PyTorch-Mojo integration lacks synergy because Mojo replaces PyTorch frontend
“Mojo as a fundamental frontend would be replacing PyTorch, not, like, augmenting PyTorch. So, in that sense, I don't see a synergy in more deeply, like, integrating Mojo.”
Chintala: Apple's MLX will fail server-side due to lack of differentiation
“If they end up expanding onto the server side, and they'll probably build something like PyTorch as well, right? Like, eventually, that'll where it will land. And I think there, they will kind of fail on the, like, lack of differentiation. Like, it wouldn't be…”
Chintala: Nvidia's primary competitive moat is NVLink interconnect, not GPU silicon
“The mode that Nvidia has right now, I feel like, is that they're, they have the interconnect that no one else has. Like, AMD GPUs are pretty good. I'm sure there's very silicon that is not bad at all, but, like, the interconnect like, NVLink is uniquely awesom…”
Chintala: LLM inference market will become a low-margin laundromat business
“My view of the LLM inference market in general is that it's like the laundromat model. Like you, the margins are going to drive down towards the bare minimum, like It's gonna be all kinds of arbitrage between how much you can get the hardware for, and then how…”
Chintala: Inference Moats from Fast CUDA Kernels Last Only Months
“I think, like, Together and Fireworks and all these people are trying to build some faster CUDA kernels and faster, like, you know, hardware kernels in general. But those modes only last for a month or two. Like, these ideas quickly propagate.”
Chintala: PyTorch is used in Mars rover simulations, drug discovery, and Tesla
“It's used in Mars rover simulations, to drug discovery, to Tesla cars, and there's a huge diversity of, like, applications in which it is used in.”
Chintala: CERN uses PyTorch and GANs for particle physics research
“I think the scariest was when I went to visit CERN at some point, and they said they were using it, PyTorch, and they were using GANs at the same time for, like, particle physics research, and I was scared more about the fact that they were using GANs than the…”
Chintala: Synthetic data only works where humans already have symbolic models
“Outside of this, like, where we don't have good symbolic models, like, synthetic data obviously, like, doesn't make any sense. So synthetic data is not a magic wand where it'll work in all cases, in every case, you know, whatever. It's just where we as humans …”
Chintala: Meta will have over 600k H100 GPU equivalents by end of 2024
“That is by the end of this year, and 600 K H-One hundred equivalents. With 250 K H-one hundreds and including all of the other GPU or accelerator stuff, it would be 600 and something K aggregate capacity.”
Chintala: Time and data constrain Meta LLM releases more than GPUs
“So, I think the, it's all a matter of time. I think time is the biggest bottleneck. It's like, when do you stop training the previous one, and when do you start training the next one? And how do you make those decisions? The data, do you have net new data, bet…”
Chintala: No AI company currently feels they have sufficient compute
“If you don't have enough compute, you figure out how to make do with smaller models, but like, no one as of today, I think, would feel like they have enough compute. I don't think, like, I've heard any company within the AI space be like, oh yeah, like, we fee…”
Chintala: Hyperscalers build custom silicon to exploit vertical workload efficiencies
“Each large company has a sufficient enough set of verticalized workloads that have a pattern to them that, say, a more generic accelerator like an NVIDIA or an AMD GPU does not exploit. So there is some level of power efficiency that you're leaving on the tabl…”
Chintala: Meta built FAIR because AI capabilities rate-limited Meta's product development
“And for, then, like, the thesis was very simple. It was, like, AI is currently rate-limiting Meta's ability to do things. Our ability to build various product integrations, moderation, various other factors. Like, AI was the limiting factor, and we just wanted…”
Chintala: Childhood trust in institutions determines views on open AI safety
“And this might be a little controversial, but like, I find a lot of arguments, ah, based on whether, like, closed source models are safer, or open source models are safer, very much related to whether, what kind of cultural, ah, culture they grew up in, what k…”
Chintala: Aggregate open source AI model usage rivals GPT
“Maybe open source models are being as used as GPT is at this point in, like, all kinds of, in a very fragmented way. Like, in aggregate, all the open source models together are probably being used as much as GPT is. Maybe, you know, close to that.”
Chintala: Less than 1% of open source AI usage yields feedback
“But the amount of feedback that is driving back into the open source ecosystem is, like, negligible. Maybe less than one percent of, like, the usage.”
Chintala: Centralized feedback could trigger open source runaway over OpenAI
“If that central sinkhole is there, who's gonna go coordinate all of this integration across all of these, like, open source frontends? But I think if we do that, if that actually happens, I think that probably has a real chance of the open source models having…”
Chintala: Open source cannot beat Google's feedback distribution advantage
“Probably doesn't have a chance against Google, because, you know, Google has Android, and Chrome, and Gmail, and Google Docs, and everything, you know. So people just use that a lot.”
Chintala: Path to AGI will be continuous, not a sudden threshold
“The whole process of, like, how we think we got to AGI will be continuous and not, like, not discontinuous”
Chintala: Open source will reach AGI via pure natural selection
“The open source thing will be very much in line with getting to AGI, because open source has that, like natural selection effect. Like, if a better open source model comes, Really no one says, huh, I don't want to use it because there are ecosystem effects, I'…”
Chintala: LMSYS leaderboard is biased and misses major use cases
“The LLSS leaderboard is the best thing we have right now to understand whether a model is better or not versus another model, but it's also biased and only having a sliver of view into how people actually use these models. Like, the people who actually end up …”
Chintala: Home robotics is five to seven years away from commercialization
“Home robotics being, like, five to seven years away into commercialization. I think, like, It's not, like, next year or two years from now, but, like, five to seven years from now, I think, like, a lot more robotics companies might pop out.”
Chintala: Robotics requires engineering work rather than fundamental breakthroughs
“My view is actually hardware is still the bottleneck, and AI is also a little bit of bottleneck, but, like, I don't think there's any, like, obvious breakthroughs we need. I think it's just work.”
Chintala: NYU robotics team deployed home robots into dozens of NYC apartments
“A couple of months ago we deployed Our home robotics stuff into, like, several tens of New York City homes, and like, try to make it do a bunch of tasks”
Chintala: Smell and touch digitization is where images were in 1920
“When we think about audio, or images, or video, they're, like, so advanced that we have the concept of color spaces, we have the concept of, like, frequency spectrums, like, you know, we figured out how ears process, like frequencies in mouse spectrum, or what…”
Chintala: In 50 years, remote smell will accompany digital media
“50 years from now, it would be pretty obvious to, like, kids of the generation to just, like, you know, I guess I was saying, I was gonna say roll a reel on their phone, maybe phones will be, they're just like, you know, on their glasses, they're watching some…”