Jan 22, 2026 · 1h 4m · mad
The End of GPU Scaling? Compute & The Agent Era — Tim Dettmers (Ai2) & Dan Fu (Together AI)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
This episode of The MAD Podcast features host Matt Turck in conversation with AI researchers Tim Dettmers and Dan Fu as they debate whether GPU scaling has hit physical hardware limits or if computational efficiency and AI agents will drive the next era of AGI development. Together, they explore hardware bottlenecks, agent management strategies, post-training workflows, and industry predictions heading into 2026.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 10.8% of the talking time here. How this is scored →
Matt as informed peer 2.8 Guest teaching 5.1 Guest disagreement 0.6 Matt pushing back 0.3
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 7:31 Tim rejects EA and rationalist AGI timelines
Tim explicitly criticizes public AGI predictions coming from effective altruism and rationalist circles as lazy thinking from living in an isolated bubble.
Hardest push from Matt ▶ 11:27 Matt directly challenges Tim on the end of GPU scalingMatt refuses to let a bold blog quote slide, directly confronting Tim with his statement that GPUs will no longer improve meaningfully and demanding a technical explanation.
Biggest teaching moment ▶ 14:20 Tim explains geometric memory bottlenecks and four-bit quantization limitsTim delivers an in-depth physics lecture explaining geometric DRAM access patterns, Von Neumann bottlenecks, and information-theoretic limits showing four-bit quantization is the end of precision scaling.
Matt holds his own ▶ 25:30 Matt unifies opposing guest essays around economic usefulnessMatt displays strong high-level synthesis by connecting the core points of both guests' opposing essays, showing how economic usefulness is the true common ground between their positions.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Guest Introductions and Dual Academic-Industry Backgrounds | 1 | 3 | 0 | 0 | Matt welcomes guests Tim Dettmers and Dan Fu and prompts them to detail their dual academic-industry backgrounds. The guests explain specialized topics like model quantization and GPU kernel optimization in an approachable manner. | |
| Defining AGI and Evaluating Economic Productivity | 2 | 4 | 1 | 0 | Matt asks both guests for a working definition of AGI. Dan and Tim gently reframe AGI away from sci-fi tropes toward tangible economic productivity and industrial impact. | |
| Tim Dettmers on Physical Limits and GPU Bottlenecks | 3 | 7 | 2 | 1 | Tim critiques superficial AGI timelines from rationalist communities and delivers a dense technical overview of physical compute limits, geometric memory constraints, and 4-bit quantization ceilings. Matt follows along closely by quoting key statements from Tim's blog post. | |
| Dan Fu on Compute Growth and Lagging Model Capabilities | 3 | 6 | 3 | 2 | Dan directly counters Tim's physical limit argument by pointing out low hardware utilization (20% MFU) and upcoming 100x compute gains from Blackwell clusters. Matt highlights Dan's core insight that current AI models are lagging indicators of hardware capabilities. | |
| Post-Training Workflows and Real-World AI Utility | 5 | 4 | 1 | 1 | Matt introduces post-training workflows and skillfully synthesizes both guests' opposing essays around practical economic utility rather than abstract AGI definitions. Both guests agree, expanding on technological diffusion and self-driving autonomy inflection points. | |
| Multi-Chip Architectures, Custom ASICs, and Local Inference | 4 | 5 | 0 | 0 | Matt demonstrates industry knowledge by bringing up alternative AI chip makers like Groq and Cerebras. Dan details low-level software abstraction challenges across AMD versus NVIDIA and distinct demands of training versus inference. | |
| The Agent Era and the Coding Singularity | 4 | 5 | 1 | 0 | Matt prompts the guests on whether AI agents have reached an inflection point, citing Tim's writing. Dan shares how Cursor agents conquered complex C++ GPU kernel writing, while Tim explains why code execution serves as a universal interface for digital tasks. | |
| Non-Coders and Building Tools with AI Agents | 2 | 5 | 0 | 0 | Matt turns to practical advice for non-programmers seeking to automate daily tasks. Tim shares a story about writing a custom video-slicing script in 20 minutes and outlines an industrial automation methodology for evaluating true task ROI. | |
| Managing AI Agents Like Junior Engineers | 2 | 5 | 0 | 0 | Matt asks Dan for management principles when using AI agents in technical workflows. Dan compares agent interaction to onboarding junior interns and explains why domain expertise dramatically increases agent leverage. | |
| Onboarding Junior Engineers in the Agent Era | 3 | 6 | 1 | 0 | Matt asks a sharp question about how junior engineers develop core domain expertise when entry-level tasks are automated. Dan shares Together AI's training approach, while Tim details the computer science education paradox. | |
| Current AI Projects at Ai2 and Together AI | 1 | 5 | 0 | 0 | Matt asks both guests to share their current research projects. Tim announces an upcoming Ai2 release enabling local 32B models to adapt to private codebases at 100x lower synthetic data cost, while Dan discusses Together AI's work on inference efficiency. | |
| Deep Dive into Mega Kernels and Together Atlas | 3 | 6 | 0 | 0 | Matt specifically asks Dan to explain Mega Kernels and Together Atlas. Dan explains how compiling an entire neural network into a single GPU kernel yields 2-3x speedups and how adaptive speculative decoding optimizes model response times over time. | |
| Predictions and Expectations for AI Progress Through 2026 | 2 | 5 | 0 | 0 | Matt asks for specific 2026 AI predictions. Tim foresees frontier model stagnation alongside the rise of specialized 100B local models, while Dan highlights new hardware generations like NVIDIA Rubin and rapid multimodal progress. | |
| Post-Transformer Architectures and Alternative Models | 4 | 5 | 0 | 0 | Matt demonstrates solid technical understanding by asking about alternative architectures like state-space models and JEPA. Dan explains how hybrid architectures and Chinese research labs are expanding model diversity beyond standard transformers. |
Statements from this episode (22)
Fu: Current LLMs meet the definition of AGI from 5-10 years ago
“By almost any definition anyone could have written down, let's say five years ago or 10 years ago, certainly when, you know, Tim, you and I started our PhD. We basically have the vision of AGI that, that we had back then. We have things that can write code. Th…”
Fu: Next-generation models currently in training will achieve AGI
“You know, we maybe already have AGI or like some form of AGI. And if not, then certainly the next generation of models, the models that today are training already. If they're at all better than what we have today, then we're, we we've already hit something tha…”
Dettmers: AGI economic adoption may temporarily dip productivity before boosting it
“And similar to when computers were introduced, productivity increased. Not initially, productivity actually went down. And you need to do diffusion in the economy to pick it up again. We might see something like that with AGI more broadly, and starts with soft…”
Dettmers: AI hardware has maxed out and won't get faster
“The hardware is maxed out. We have no new technology. We can make it easier to manufacture and a little bit cheaper, but not faster. And we have maxed out on the additional features.”
Dettmers: 4-bit precision is the end of quantization
“Four bit precision is the end of quantization.”
Dan Fu: DeepSeek-V3 was trained on ~2,000 H800s with 20% MFU
“If you look at the deep seek model, for instance, this is one of the best open source models we have out there today. It was trained at the end of 2024. On last generation, kind of nerfed GPUs, H 800 instead of H 100, the 800 is nerfed by all sorts of ways fro…”
Poolside and Reflection are building clusters with massive B200 GPU deployments
“They're companies like Poolside. They're building out tens of thousands of B-two hundred, GB-two hundred chips. You know, there's other folks like Reflection who are who are building out. Tens of thousands of B 200 chips.”
Dan Fu: Deployed AI models lag cluster infrastructure by 1–2 years
“The models that we see today that we can play with today have been pre-trained on clusters that were built out a year or two ago. Because, you know, you need enough time to get the cluster running. You need enough time to do the large pre-training run. And the…”
Fu predicts increasing hardware diversity, particularly for AI model inference
“I'm sure NVIDIA will still do great and still grow beyond their five trillion dollar company or whatever it is at the time of recording. But I think you're going to see a lot more diversity especially around, I think inference of the model.”
Fu: AI coding tools enable expert programmers to move 10x faster
“But if you give an expert programmer This set of tools, they can go 10, 10 times faster than they were able to go before.”
Dettmers: Coding agents serve as general-purpose AI agents for digital tasks
“Coding agents are general agents. Coding agents can write programs that solve other problems, and code is so general, if there's a digital problem, you could solve it for code, and coding agents make the thing so easy that now you can solve a variety of proble…”
Dettmers: Automating calendar scheduling with AI agents yields minimal productivity gain
“An agent doesn't know that. And if you specify that to an agent, you could also just create the calendar invite, the meeting pipeline, and it doesn't increase productivity by much.”
Dan Fu: Junior engineers using AI agents communicate better and level up faster
“When they are really gung ho about understanding and being able to use the AI agents, there's, they're able to communicate so much better than in the olden days. They're able to level up their level of understanding a lot faster.”
Dettmers: Students using AI agents perform poorly on basic domain knowledge
“If we let people use agents, they perform very poorly on basic knowledge. And if we let people just do the basic knowledge, they don't know how to use agents and they can't compete. So they can't do useful work in the workforce nowadays.”
Dettmers: Humans will work on problems only AI agents understand
“In the future it's realistic that we work on problems that we don't understand, that agents understand, but we need to keep up in some way”
Ai2 to release open-source coding agent with 100x cheaper training
“We will have a major release of an open source coding agent that has a couple of key features. For one, training is a hundred times cheaper. You need to generate synthetic data and you need to train on it. And so we have a method that's a hundred roughly a hun…”
Fu: Hardware utilization during AI inference is under 5%
“At inference time, when the, when you have the model, when it's already been trained, already been post-trained, the hardware utilization is like less than five percent.”
Dan Fu: Proper speculative decoding yields 2x to 3x model speedups
“So if you do the speculative decoding right, you can get, again, two X, three X speed ups over, over, you know, just running a vanilla model.”
Dettmers: No open-source system supports complex inference across eight-plus GPUs
“There's no open source system that can do that at the moment.”
Dettmers: Frontier AI performance will stagnate while smaller models improve
“Performance on the frontier will stagnate, but on the smaller level, we get more and more powerful models still, because you can distill from these large models into these small models.”