ULMFiT
topic on 1 show · 4 statements across 1 episodes · said 20 times in 3 episodes since 2023
Mentions by year, every show
tap a year for its mentions
Latent Space 20
2024 7 mentions in 2 episodes 4 per episode
2023 13 mentions in 1 episode
every mention on every show, scene by scene, with the transcript →
4 statements about ULMFiT, every show
Alignment tax happens because models are fine-tuned instead of continued pre-trained
“ULM fit is the wrong approach and that's why we're seeing a lot of these You know, so-called alignment tax, and this view of like, oh, a model can't both code and do other things. You know, I think it's actually because people are training them wrong.”
Jeremy Howard: The three-step ULMFiT fine-tuning approach is wrong and obsolete
“Even though I originally created the three-step approach that everybody now does, my view is it's actually wrong, and we shouldn't use it.”
Howard: Alec Radford Built OpenAI's GPT After Reading ULMFiT
“I organized a chat for both of us with Kate Metz in the New York Times, and Kate Metz answered, sorry, and Alec answered this question for Kate, and Kate just like, so how did, you know, GPT come about? And he said, well, I was pretty sure that pre-training on…”
Howard: Modern LLMs like ChatGPT still follow ULMFiT's three-step training framework
“So we generated this three-step system. So step one was train a language model on a big corpus. Step two was fine-tune a language model on a more curated corpus, and step three was further fine-tune that model on a task. And of course, that's what everybody st…”