Mid Training
topic on 3 shows · 4 statements across 4 episodes
Latent Space 
the MAD Podcast 
the a16z Podcast
4 statements about Mid Training, every show
Kant: Mid-training stages exist due to organizational silos, not necessity
“Mid-training exists because there's a mid-training team now, right? There's people or like people decide to focus on like a mid-training effort. But what you really want is engineering. And skill of experiments that allows for a much more continuous spectrum t…”
Soldaini: Mid-training requires re-mixing pre-training data to avoid model forgetting
“When you do that, you also need to make sure that The model doesn't forget stuff that I've seen during pre-training, so that's why, like, you mix some of the best data from pre-training, you do carry over.”
Kim: Mid-training updates AI model knowledge without requiring full pre-training runs
“We do it before after pre-training, but before post-training you kind of think of a way to like extend the model's like intelligence without having to do a whole new pre-training run. So this is mostly just focused on data and off of the pre-training models. S…”
Brown: OpenAI models undergo mid-training and post-training before release
“For open AI models, like, they go through a mid-training step, and then they go through a post-training step, and then they're released, and they're a lot more useful. Like, frankly, if you interacted with the only pre-trained model, it would be super difficul…”