AI Training Run
topic on 3 shows · 4 statements across 4 episodes
Latent Space 
the MAD Podcast 
the a16z Podcast
4 statements about AI Training Run, every show
Balaban: Backwards pass accounts for two-thirds or more of training compute
“Generally speaking, when you're doing a training run, you might think that you might be some sort of split between the backwards pass and the forward pass on the model. And the backwards pass might be, let's say, two thirds or more of the compute and the forwa…”
Casado: A $1B model training run economically justifies a custom ASIC
“No, no, no, a billion dollar training run, a one billion dollar training run, it makes sense to actually do a custom ASIC if you can do it in time. The question now is timeline. Not money. Cause just rough math. If it's a billion-dollar training run, then the …”
Andreessen: AI model training runs can be paused and resumed seamlessly
“Training runs can be paused, right? You can actually like, you can shut a training run off for two weeks and then start it up again. Which, you know, you can't do if you're running a Gmail service or a search engine or something.”
Pesenti in 2020: Multi-million-dollar AI training runs are unsustainable for Facebook
“Where it becomes millions is when you do training runs. So some of the training runs in the most advanced system that it comes from our company or other companies out there are starting to be extremely expensive. Yeah. Like you can look at one run in the scale…”