FlashAttention
product on 6 shows · 8 statements across 5 episodes · said 63 times in 24 episodes since 2023
Latent Space 53 
No Priors 3 
the MAD Podcast 3 
Invest Like the Best 2 
the Y Combinator Startup Podcast 1 
Sourcery 1
Mentions by year, every show
tap a year for its mentions
Latent Space 53the MAD Podcast 3No Priors 3Invest Like the Best 2Sourcery 1the Y Combinator Startup Podcast 1
2026 6 mentions in 4 episodes 2 per episode
- Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper
- The End of GPU Scaling? Compute & The Agent Era — Tim Dettmers (Ai2) & Dan Fu (Together AI)
- Multi-GPU Kernels, Intelligence per Watt, Heterogeneous Inference, and More | YC Paper Club · Y Combinator
- NVIDIA’s Jensen Huang on Reasoning Models, Robotics, and Refuting the “AI Bubble” Narrative
- every mention in 2026, scene by scene →
2025 19 mentions in 4 episodes 5 per episode
- ⚡️ Beyond Transformers with Power Retention
- How Zyphra went all-in on AMD + Why Devs feel faster with AI but are slower — with Quentin Anthony
- The Shape of Compute (Chris Lattner of Modular)
- ⚡️Mercury: Ultra-Fast Diffusion LLMs — Estefano Ermon, CEO Inception Labs
- every mention in 2025, scene by scene →
2024 15 mentions in 10 episodes 2 per episode
- Building an open AI company - with Ce and Vipul of Together AI
- llm.c's Origin and the Future of LLM Compilers - Andrej Karpathy at CUDA MODE
- Answer.ai & AI Magic with Jeremy Howard
- The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert
- 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
- [Paper Club] Upcycling Large Language Models into Mixture of Experts
- Language Agents: From Reasoning to Acting — with Shunyu Yao of OpenAI, Harrison Chase of LangGraph
- Computer Science is 'So Hot Right Now' | Grace Isford, Lux Capital · Sourcery with Molly O'Shea
- 2 more episodes that year, every mention in 2024 →
2023 23 mentions in 6 episodes 4 per episode
- FlashAttention-2: Making Transformers 800% faster AND exact
- The End of Finetuning — with Jeremy Howard of Fast.ai
- Ep 18: Petaflops to the People — with George Hotz of tinycorp
- No Priors Ep. 9 | With Perplexity AI’s Aravind Srinivas and Denis Yarats
- Guardrails AI: The Playbook for Safer, Hallucination-Free LLMs — Shreya Rajpal Explains
- RWKV: Reinventing RNNs for the Transformer Era
- every mention in 2023, scene by scene →
every mention on every show, scene by scene, with the transcript →
8 statements about FlashAttention, every show
Modular's Mojo FlashAttention beats Tri Dao's reference implementation
“We're beating the tree DAO reference implementation that everybody uses, for example, right? Written fully in Mojo. Again, all of our, GPU kernels are written in Mojo. You can go see the history of the team building this, and it was done in just a few weeks, r…”
Isford: Together AI Has the Market's Fastest Inference Tech
“They have the author of Flash Attention, Sri Dao, who is brilliant, as their chief scientist. He is, has a proprietary version of the fastest, like, inference technology on the market that's low cost to serve.”
Zhang: ML systems optimization is fundamentally always about data movement
“Fundamentally, the thing we are actually optimizing is actually not that different. It's always about data movement across essentially all the stacks, right? So when you do distributed, like computing, it's about communication across different machines. When y…”
Howard: Mojo-like languages will unlock thousands of FlashAttention-scale breakthroughs
“There is a thousand flash attentions out there for us to build. You just got to make it easy for us to build them. So like Triton definitely helps. But it's still, Not easy. You know, it still requires kind of really understanding the VPU architecture, writing…”
FlashAttention achieves 2x to 4x wall-clock speedup with linear memory
“So in the end, we ended up being, the memory is linear in sequence length. In terms of computation, it's still quadratic, but we managed to make it much more hardware friendly, and as a result, we do get wall clock speed up on the order of two to four X which …”
Kernel fusion sacrifices flexibility for researchers experimenting with attention
“When you do kernel fusion is a little bit you lose a little bit of flexibility in the sense that, hey, now you have for example, is flash attention is just a subroutine that you would call to do attention. But as a researcher, let's say you don't want that exa…”
Releasing highly optimized code mattered more than the FlashAttention paper
“I think when we were writing the paper, I remember sending an email to one of my advisors, like hey, I'm excited about this paper but I think the most important thing will be the artifact, which is the code. So I knew that, like, the code will be valuable and,…”
FlashAttention techniques generalize across accelerators with asymmetric memory
“I expect the idea to be broadly these ideas to be broadly applicable to different hardware. As long as, I think the main idea is you have, like, asymmetry in, in memory hierarchy, which tends to be everywhere, you know, in, in a lot of a lot of accelerators.”