Nguyen: Post-training scaling avoids data walls through infinite learnable tasks
Karina Nguyen · OpenAI researcher on why soft skills are the future of work | Karina Nguyen · Feb 9, 2025 · at 9:56
OpenAI researcher Karina Nguyen responds to arguments that artificial intelligence progress is plateauing due to running out of raw internet pre-training data.
0:00 / 0:43exact quote · 43.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The scaling in post-chaining itself is not hitting the wall, and that's because Basically, we went from, like, raw data sets from pre-trained models to infinite amount of tasks that you can teach the model in the post-training world via reinforcement learning. So any task, for example, like, how to search the web, how to use the computer, how to write, ah, well, like, all sorts of tasks that you, like, trying to teach the model All the different skills. And that's why we're saying, like, there's no data wall or whatever, because there will be infinite amount of tasks.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Karina Nguyen
Nguyen: AI bottleneck is evaluations rather than data as benchmarks saturate
“We are actually getting saturated in all benchmarks. So I think the bottleneck is actually in evaluations that we don't have all the frontier, like evals, like, I don't know GPGA, which is, like, A Google-proof question answering, like, PhD-level intelligence …”
Nguyen: Synthetic Data Outperforms Human Data for AI Product Development
“And the reason why I really love, like, synthetic, like, relying purely on synthetic data instead of, like, collecting Data from humans is because it's, like, much more scalable. It's cheap, less than how, like, you literally sample from the model, and you tea…”
Nguyen: ChatGPT still struggles with writing due to creative reasoning limits
“I think it's actually really, really hard to teach the model how to be aesthetic or, like, do, like, visual, really good, like, visual design or, like, how to be extremely creative in the way they write. I think, like, I still think, like, Chai GP kind of suck…”
Nguyen: AI research progress is bottlenecked by research management
“I actually, like, AI research progress is bottlenecked by, like, management. Like, research management is because you have, like, constrained set of compute, and you need to, like, allocate the compute to the research path that you feel the most Commenced abou…”
Nguyen: AI is not far from autonomous self-improving product development
“And I don't think, like, we are far away from that kind of, like, self-improvement, models becoming, like, self-improved via, like, then, like, the product development is basically kind of, like, self-improving, like, it's kind of, like, its own, like, organis…”
Nguyen: Anthropic excels at prioritization, while OpenAI takes more product risks
“I would say, like, Antarctic, I learned from Antarctic that, like, They're much better at, like, focusing and, like, prioritization or, like, very, very hard, like, very hardcore prioritization, I guess, and they need to do it. Like, but I think, like, OpenAI …”