Dan Fu: Deployed AI models lag cluster infrastructure by 1–2 years
Dan Fu · The End of GPU Scaling? Compute & The Agent Era — Tim Dettmers (Ai2) & Dan Fu (Together AI) · Jan 22, 2026 · at 21:26
Dan Fu explains why visible AI model performance lags behind current GPU hardware capabilities.
0:00 / 0:19exact quote · 19.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The models that we see today that we can play with today have been pre-trained on clusters that were built out a year or two ago. Because, you know, you need enough time to get the cluster running. You need enough time to do the large pre-training run. And then you need enough time to really post train it, fine tune it, do all the RLHF and all that stuff.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Fu
Fu: Current LLMs meet the definition of AGI from 5-10 years ago
“By almost any definition anyone could have written down, let's say five years ago or 10 years ago, certainly when, you know, Tim, you and I started our PhD. We basically have the vision of AGI that, that we had back then. We have things that can write code. Th…”
Fu: Next-generation models currently in training will achieve AGI
“You know, we maybe already have AGI or like some form of AGI. And if not, then certainly the next generation of models, the models that today are training already. If they're at all better than what we have today, then we're, we we've already hit something tha…”
Fu: AI coding tools enable expert programmers to move 10x faster
“But if you give an expert programmer This set of tools, they can go 10, 10 times faster than they were able to go before.”
Dan Fu: DeepSeek-V3 was trained on ~2,000 H800s with 20% MFU
“If you look at the deep seek model, for instance, this is one of the best open source models we have out there today. It was trained at the end of 2024. On last generation, kind of nerfed GPUs, H 800 instead of H 100, the 800 is nerfed by all sorts of ways fro…”