Habib: Next-word prediction at scale forces LLMs to develop reasoning
Raza Habib · The REAL potential of generative AI · Y Combinator · Feb 28, 2023 · at 2:55
Humanloop CEO Raza Habib explains to YC partner Ali Rowghani why predicting the next word eventually forces AI models to learn reasoning and facts.
0:00 / 0:07exact quote · 7.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“They are able to do this task extremely well. And the only way to do that is to have gotten better at under, you know, some form of reasoning and some form of knowledge.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Raza Habib
Habib: Senior developers benefit more from GitHub Copilot than junior developers
“One thing that is surprising to me is that the people who say to me they use it the most are some of the people I consider to be better or more senior developers. You might've thought this tool would help juniors more, but I think people who are more accustome…”
Habib: Software engineers will be among the first jobs automated by AGI
“When we do get towards things that look like AGI, I suspect that developers will actually be one of the first jobs to see large fractions of their job be automated, which I think is very counterintuitive but also predicting the future is hard, so.”
Habib: Foundation model barriers are capital and talent, not secret sauce
“Like to me, the barriers to entry of training, one of these models are mostly capital and talent. Like the people needed are still very specialized and very smart and you need lots of money to pay for GPUs. But beyond that, I don't see that much secret sauce, …”
Habib: Skeptical feedback data gives general model creators an insurmountable flywheel
“There's some question about, you know, whether or not the feedback data might give them a flywheel. I'm a little bit skeptical of that, that it would give them so much that no one could catch up.”
Habib: Scaling LLM parameters and data consistently improves prediction accuracy
“As you scale the language models, both in terms of the number of parameters they have, but also in the size of the data set that they're trained on, it turns out that they continue to get better and better at this prediction task.”
Habib: ChatGPT's obsequious tone drove the need for custom, use-case-specific models
“I think when ChatGPT came out, there was a lot of frustration from people who didn't like its personality. The tone was a bit obsequious and it's, you know, it'll defer. It doesn't want to give strong opinions on things. And to me that demonstrates the need fo…”