LLM Training
topic on 3 shows · 3 statements across 3 episodes
Latent Space 
the MAD Podcast 
the a16z Podcast
3 statements about LLM Training, every show
Cubuk: Physical experiments provide unhackable training signals for LLMs
“As we measure it, the LLM will get very clear signal. It's hard to hack, you know, unless, unlike these other LLM training techniques, it's like really what you see in real life is the signal that's going to the LLM.”
Agarwal: Distilling a Large Model Outperforms Direct Training on the Same Data
“This is something that people have found again and again, that basically you can train a model on some data, or you can train a bigger model on that data and distill that model to another model, and that distill model is better.”
Socher: Don't train LLMs from scratch without hundreds of millions of responses
“And yeah, I don't think you should attempt it unless you have, you know, hundreds of millions of responses that to really like train your own model versus just fine-tune it a little bit. Like certainly not from scratch.”