Model Collapse
topic on 3 shows · 5 statements across 4 episodes
Latent Space 
the MAD Podcast 
20VC
5 statements about Model Collapse, every show
Anchoring self-improvement loops with real external signals prevents AI model collapse
“Model collapse mainly happens when you have a loop that is Completely closed. Right. And if you don't have any outside signal and just the model, for example, talking to itself or operating in a very like a restricted environment there's a good chance that you…”
Pineau: Synthetic data for code avoids model collapse through injected diversity
“If I think of coding, we can generate synthetic code. You take normal code and we know how to inject diversity into the code. Like I can take a couple of repositories, mix and match, apply an LLM to transform it. And so there's a way to generate synthetic data…”
Morcos: Filtering synthetic data between generation cycles prevents model collapse
“If you filter the data after each point, that's now information injection, and that can break all of this and I think can prevent model collapse.”
Ben Allal: Properly curated synthetic data prevents model collapse
“And I think there's a lot of concerns about model collapse, and I'm going to talk about that later, but we'll see that like, if we use synthetic data properly and we curate it carefully that shouldn't happen.”
Ben Allal: Distilling from large models avoids small-model self-training collapse
“I think if you do that approach, it's normal to observe this kind of behavior because the quality is going to be worse because the model is already small, and then if you train it just on these generations, you shouldn't expect it to become better. But what we…”