Agentic AI workflows are technically feasible today but face reliability hurdles
Ryan Glasgow · A guide to building product in a post-LLM world | Ryan Glasgow and Kevin Mandich from Sprig · Sep 7, 2023 · at 1:14:55
Sprig CEO Ryan Glasgow responds to whether autonomous agent workflows (like AutoGPT) for product management tasks are achievable in the near term.
0:00 / 0:23exact quote · 23.0s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Based on what we've seen today in the boundaries of the LLMs today is that we could, based on what we have tested with the models, actually build that now. And we are seeing the sophistication, the capability of the models to build that now. I think the roadblocks we've seen is more around the reproducibility, the reliability, you know, some of the other edge cases like the quantitative analysis. And I think the other piece is really the comfort of the user.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ryan Glasgow
Defining correctness in AI outputs is becoming increasingly subjective
“As we, I think, trend towards artificial general intelligence, the definition of correct or the definition of whether something is, you know, valid or not, It's becoming increasingly subjective”
Much of AI companies' value now lies in prompt engineering
“And I think a lot of the value for AI companies now, you know, is in prompt engineering. I think the early days was actually just Can you build a model? Can you get the model running? Can you get the model working in production? Where I think a lot of it is, c…”
ChatGPT succeeded partly by remaining a standalone product outside core workflows
“And when you look at ChatGPT, I think partly why it's been successful is that it's such a standalone product that's not integrated into anything else. And I think if it actually was integrated into your Slack or into your email or into your work, I think peopl…”
Fast AI evolution requires hiring for learning slope over specific domain experience
“Making sure that the slope is there is really critical as someone who's really smart, just because the AI field is changing and particularly this year, every week, it's rapidly changing. And so you're going to need someone who you can't rely on their prior exp…”
Three-worker Mechanical Turk consensus often failed expert data accuracy checks
“We originally hired Amazon Mechanical Turks, and we said, hey, we'll save money. We'll have three of them review every response. But we'd often see that even if all three of them gave the same output, we would go to an expert researcher or a customer or look a…”
AI models will likely never reach total accuracy for subjective feedback categorization
“Cause we will never, it's very unlikely we'll ever get to a hundred percent. And so we'll get, you know, 90, 95, 99. But we'll always make sure that they have the ability to then do that last mile of analysis on their own. And so that's something that we have …”