ULMFiT
4 statements across 1 episodes · 1 bullish · 2 bearish · 1 people on the record · first statement Oct 20, 2023 by Jeremy Howard · said 20 times in 3 episodes since 2023 · across every show →
Mentions by year
brought up most by Jeremy Howard (9)
tap a year for its mentions
2024 7 mentions in 2 episodes 4 per episode
2023 13 mentions in 1 episode
Everything said about ULMFiT, oldest first
Oct 20, 2023 negative
Alignment tax happens because models are fine-tuned instead of continued pre-trained
“ULM fit is the wrong approach and that's why we're seeing a lot of these You know, so-called alignment tax, and this view of like, oh, a model can't both code and do other things. You know, I think it's actually because people are training them wrong.”
Oct 20, 2023 neutral
Howard: Modern LLMs like ChatGPT still follow ULMFiT's three-step training framework
“So we generated this three-step system. So step one was train a language model on a big corpus. Step two was fine-tune a language model on a more curated corpus, and step three was further fine-tune that model on a task. And of course, that's what everybody st…”
Oct 20, 2023 positive
Howard: Alec Radford Built OpenAI's GPT After Reading ULMFiT
“I organized a chat for both of us with Kate Metz in the New York Times, and Kate Metz answered, sorry, and Alec answered this question for Kate, and Kate just like, so how did, you know, GPT come about? And he said, well, I was pretty sure that pre-training on…”