Language Modeling
topic on 5 shows · 7 statements across 7 episodes
the Y Combinator Startup Podcast 
the Knowledge Project 
No Priors 
the MAD Podcast 
20VC
7 statements about Language Modeling, every show
Brockman: 2017 sentiment neuron proved semantics emerge from next-character prediction
“Because it's 2017, and it's really the first time that we saw semantics arise from training on A language modeling objective. So you train on learn the next character, predict the next character, and then suddenly you get a neural net that understands sentimen…”
Finn: Pre-training and curated fine-tuning unlocked robotic laundry folding
“And this was actually to take some inspiration from the world of language modeling to actually instead of just training a policy on all of our data, can we pre-train on all the data? And then fine tune on a highly, on a curated, consistent, high quality set of…”
Gomez: Google failed to lean into language modeling early, unlike OpenAI
“To say they didn't lean hard enough into language modeling, like just pure Sequence modeling of text on the internet. That's, I think the accurate statement. That's what OpenAI did early and uniquely well.”
Chip Huyen: Lack of labeled data requirements makes language modeling uniquely scalable
“You don't need to curate, like, labels, like, reference data, so that, that you can use a train models that make language modeling, like, so, so much easier to scale than other types of tasks.”
Gomez: Industry took two to three years to realize language model scaling worked
“With language modeling and the whole scaling project, I thought the world would catch on way faster to that piece. It started to become really obvious, but then it was two, three years before everyone woke up, and it sort of hit the world.”
Gu: Mamba successfully applied state-space models to language modeling
“Recently proposed a model called Mamba which was kind of brought these to language modeling and showed really good results there.”
Shazeer Calls Next-Token Prediction an AI-Complete Problem
“And like, the problem is super simple to define. It's just like predict the next word, the fat cat sat on the, you know, like, okay, well, you know, what comes next? Like it's extremely easy to define. And if you can do a great job of it, like, you know, then …”