Dan Fu: DeepSeek-V3 was trained on ~2,000 H800s with 20% MFU
Dan Fu · The End of GPU Scaling? Compute & The Agent Era — Tim Dettmers (Ai2) & Dan Fu (Together AI) · Jan 22, 2026 · at 17:26
Dan Fu, Assistant Professor at UCSD and VP of Kernels at Together AI, discusses hardware utilization efficiency during the training of open-source frontier models.
0:00 / 0:40exact quote · 40.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“If you look at the deep seek model, for instance, this is one of the best open source models we have out there today. It was trained at the end of 2024. On last generation, kind of nerfed GPUs, H 800 instead of H 100, the 800 is nerfed by all sorts of ways from NVIDIA to get around the expert restrictions at the time. And they were trained with, let's call it like about 2000 H 800 according to the report for, I think about a month. And when you compute how much, how long that took, when you see how much compute was actually available on the chip, you get something like a 20% effective chip utilization or something like that.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Dan Fu
Fu: Current LLMs meet the definition of AGI from 5-10 years ago
“By almost any definition anyone could have written down, let's say five years ago or 10 years ago, certainly when, you know, Tim, you and I started our PhD. We basically have the vision of AGI that, that we had back then. We have things that can write code. Th…”
Fu: Next-generation models currently in training will achieve AGI
“You know, we maybe already have AGI or like some form of AGI. And if not, then certainly the next generation of models, the models that today are training already. If they're at all better than what we have today, then we're, we we've already hit something tha…”
Fu: AI coding tools enable expert programmers to move 10x faster
“But if you give an expert programmer This set of tools, they can go 10, 10 times faster than they were able to go before.”
Dan Fu: Deployed AI models lag cluster infrastructure by 1–2 years
“The models that we see today that we can play with today have been pre-trained on clusters that were built out a year or two ago. Because, you know, you need enough time to get the cluster running. You need enough time to do the large pre-training run. And the…”