Nguyen: Claude Refused Setting Alarms After Realizing It Lacked a Body
Karina Nguyen · OpenAI researcher on why soft skills are the future of work | Karina Nguyen · Feb 9, 2025 · at 7:05
Karina Nguyen, former Anthropic post-training researcher now at OpenAI, describes training edge cases encountered during Claude 3 development.
0:00 / 0:35exact quote · 35.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“One of the things that I've learned early days at Anthropic was, like, we've discovered, especially with, like, cloud three training, when you taught the model some of the self-knowledge of, like, hey, like, you actually don't have a physical body to operate, like, in the physical world. But then at the same time, we had data that kind of taught the model some of the function calls, which is, like, this is how you set the alarm. And so the model would get, like, extremely confused about, like, whether it can set an alarm, but it doesn't have a body in the physical world.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Karina Nguyen
Nguyen: Post-training scaling avoids data walls through infinite learnable tasks
“The scaling in post-chaining itself is not hitting the wall, and that's because Basically, we went from, like, raw data sets from pre-trained models to infinite amount of tasks that you can teach the model in the post-training world via reinforcement learning.…”
Nguyen: AI bottleneck is evaluations rather than data as benchmarks saturate
“We are actually getting saturated in all benchmarks. So I think the bottleneck is actually in evaluations that we don't have all the frontier, like evals, like, I don't know GPGA, which is, like, A Google-proof question answering, like, PhD-level intelligence …”
Nguyen: Synthetic Data Outperforms Human Data for AI Product Development
“And the reason why I really love, like, synthetic, like, relying purely on synthetic data instead of, like, collecting Data from humans is because it's, like, much more scalable. It's cheap, less than how, like, you literally sample from the model, and you tea…”
Nguyen: ChatGPT still struggles with writing due to creative reasoning limits
“I think it's actually really, really hard to teach the model how to be aesthetic or, like, do, like, visual, really good, like, visual design or, like, how to be extremely creative in the way they write. I think, like, I still think, like, Chai GP kind of suck…”
Nguyen: AI research progress is bottlenecked by research management
“I actually, like, AI research progress is bottlenecked by, like, management. Like, research management is because you have, like, constrained set of compute, and you need to, like, allocate the compute to the research path that you feel the most Commenced abou…”
Nguyen: AI is not far from autonomous self-improving product development
“And I don't think, like, we are far away from that kind of, like, self-improvement, models becoming, like, self-improved via, like, then, like, the product development is basically kind of, like, self-improving, like, it's kind of, like, its own, like, organis…”