Ezinne Udezue: AI PMs must master evaluations, not just prompt engineering
Ezinne Udezue · How AI is reshaping the product role | Oji and Ezinne Udezue · Sep 7, 2025 · at 20:01
Ezinne Udezue discusses the hard technical skill requirements for product managers working with generative AI and large language models.
0:00 / 0:35exact quote · 35.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“There's this skill of being able to write evals. I know everybody can write prompts, prompt engineering. You can try and focus the LLM so that it can offer better insights and offer better results. But even as your LLM actually Provide, produces results. You need to verify that it actually is smart and it hasn't hallucinated or gone off by creating true evals, right? Those are some hardcore skills. Being able to constrain hallucination. Knowing what ways you do that. Being able to use multiple models.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ezinne Udezue
Ezinne Udezue: Tension between CPOs and CTOs drives innovation over combined roles
“There is a conflict between the CPO and the CTO that actually drives innovation in my opinion, because one person is trying to optimize the resourcing, the fixed, right? While another person is actually trying to field opportunities and figure out how to wait …”
Ezinne Udezue: Bolting AI onto legacy codebases will not revolutionize industries
“Companies where their code base remains and they are attaching AI to it often aren't the ones who are going to revolutionize their industry. Companies for whom their code base perhaps shrinks and the LLM becomes a core part of what it is that they use to solve…”
Ezinne Udezue: AI summaries cannot replace direct observation of customer behavior
“True customer insight comes from not Using AI to read all the transcripts of your customer interviews. That isn't what it is. It's actually trying to figure out how to best walk in their shoes and whatever it takes, you've got to figure out how to do that beca…”
Ezinne Udezue: Generate content yourself and let AI refine it, not vice-versa
“Don't, even though you keep talking about it as generative AI, be the generator and let it refine. That's the major trick. You know, that's the hack everybody should learn.”
Ezinne Udezue: AI shifts software from static artifacts to living systems
“So that idea of working static software isn't true anymore. It's almost like living, breathing software and ensuring that the things that allow it to breathe actually occur.”
Ezinne Udezue: Shipping opinionated design beats abandoning users to multiple configurations
“The reality is, and my experience has shown that you're better off picking an opinion, shipping, shipping it out there, being wrong, and then adjusting that leaving too many options because you then can never learn what was the better experience overall.”