Weil: Current AI models are eval-limited rather than intelligence-limited
Kevin Weil · A conversation with OpenAI's CPO Kevin Weil, Anthropic's CPO Mike Krieger, and Sarah Guo · Nov 6, 2024 · at 13:12
OpenAI CPO Kevin Weil discusses model progression and deployment challenges with Anthropic CPO Mike Krieger and Sarah Guo.
0:00 / 0:13exact quote · 13.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think there's a very real sense in which models today are not intelligence limited. They're eval limited. They can actually do much more and be much more correct on a wider range of things than they are today, and it's really about sort of teaching them.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kevin Weil
Weil: OpenAI will not build industry-specific vertical AI products
“There are immense opportunities in every industry and every vertical in the world to go build AI based products that improve upon the state of the art. And there's just no way we could ever do that ourselves. We don't want to, we couldn't, if we did want to.”
Weil: OpenAI minimizes scaffolding because rapid advances erase model limits
“We don't spend that much time building scaffolding around the parts that don't match that because our general mindset is in two months, there's going to be a better model and it's going to blow away whatever, you know, the current set of limitations are.”
Weil admits OpenAI no longer holds a massive 12-month lead
“It used to be that OpenAI had this, like, massive model lead, you know, 12 months or something ahead of everybody else. That's not true anymore. You know, I like to think we still have a lead. I'd argue that we do, but it's certainly not a massive one.”
Weil: Organizations should stay PM-light to prevent micromanagement
“Too many PMs causes problems. You know, we'll like fill the world with decks and ideas versus execution. So I think that the, I think it's a good thing when you have a PM that has that is working with maybe slightly too many engineers, because it means that th…”
Weil predicts ML engineers will soon embed in every product team
“And so I think you're going to want sort of quasi researcher machine learning engineer types as part of pretty much every team because fine tuning a model is just going to be part of the core workflow for building most products.”
Weil: OpenAI manages 400 million weekly users with 30-40 support staff
“With 400 plus weekly 400 plus million weekly active users, we get, you know, a lot of inbound tickets, right? I don't know how many customer support folks we have, but it's not very many. 3040, I'm not sure. Way, way smaller than you would have at any comparab…”