Defining correctness in AI outputs is becoming increasingly subjective
Ryan Glasgow · A guide to building product in a post-LLM world | Ryan Glasgow and Kevin Mandich from Sprig · Sep 7, 2023 · at 13:27
Ryan Glasgow, founder and CEO of Sprig, reflects on customer reactions to AI-generated qualitative research findings.
0:00 / 0:10exact quote · 10.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“As we, I think, trend towards artificial general intelligence, the definition of correct or the definition of whether something is, you know, valid or not, It's becoming increasingly subjective”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ryan Glasgow
Much of AI companies' value now lies in prompt engineering
“And I think a lot of the value for AI companies now, you know, is in prompt engineering. I think the early days was actually just Can you build a model? Can you get the model running? Can you get the model working in production? Where I think a lot of it is, c…”
ChatGPT succeeded partly by remaining a standalone product outside core workflows
“And when you look at ChatGPT, I think partly why it's been successful is that it's such a standalone product that's not integrated into anything else. And I think if it actually was integrated into your Slack or into your email or into your work, I think peopl…”
Agentic AI workflows are technically feasible today but face reliability hurdles
“Based on what we've seen today in the boundaries of the LLMs today is that we could, based on what we have tested with the models, actually build that now. And we are seeing the sophistication, the capability of the models to build that now. I think the roadbl…”
Fast AI evolution requires hiring for learning slope over specific domain experience
“Making sure that the slope is there is really critical as someone who's really smart, just because the AI field is changing and particularly this year, every week, it's rapidly changing. And so you're going to need someone who you can't rely on their prior exp…”
Three-worker Mechanical Turk consensus often failed expert data accuracy checks
“We originally hired Amazon Mechanical Turks, and we said, hey, we'll save money. We'll have three of them review every response. But we'd often see that even if all three of them gave the same output, we would go to an expert researcher or a customer or look a…”
AI models will likely never reach total accuracy for subjective feedback categorization
“Cause we will never, it's very unlikely we'll ever get to a hundred percent. And so we'll get, you know, 90, 95, 99. But we'll always make sure that they have the ability to then do that last mile of analysis on their own. And so that's something that we have …”