Sep 7, 2023 · 1h 16m · in-depth
A guide to building product in a post-LLM world | Ryan Glasgow and Kevin Mandich from Sprig
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Sprig founder Ryan Glasgow and Head of Machine Learning Kevin Mandich share tactical guidance on building AI-first products, detailing their evolution from early human-in-the-loop NLP systems to modern LLMs and autonomous agent workflows. They explore essential strategies for hiring ML talent, embedding AI engineers into product development triads, preserving human agency, and establishing defensibility in a rapidly shifting technology landscape.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Brett holds 16.4% of the talking time here. How this is scored →
Brett as informed peer 4.3 Guest teaching 3.9 Guest disagreement 0.1 Brett pushing back 0.2
speaking balance: gold is Brett, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 49:11 Ryan counters commoditization concerns
Ryan directly refutes investor skepticism that LLMs commoditize Sprig's core differentiation by pointing to continuous innovation and proprietary data collection.
Hardest push from Brett ▶ 48:47 Brett probes risk of disruptive innovationBrett directly challenges Ryan on whether advancements in foundational LLMs make Sprig's core proprietary value proposition obsolete.
Biggest teaching moment ▶ 10:28 Kevin explains absence of ML ground truth in user researchKevin educates Brett on why traditional ML fails in UX analysis because two expert researchers will categorize the same responses completely differently.
Brett holds their own ▶ 45:37 Brett introduces Betty Crocker analogy for AI UXBrett demonstrates sharp product intuition by connecting user adoption of AI to the classic Betty Crocker egg effect where users need hands-on contribution.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Brett as informed peer | Guest teaching | Guest disagreement | Brett pushing back | Why |
|---|---|---|---|---|---|---|
| Founding Sprig: Solving Qualitative Feedback at Scale | 4 | 3 | 0 | 0 | Brett sets the stage by asking Ryan to describe the origins of Sprig and early applied ML. Ryan and Kevin articulate the fundamental problem of analyzing messy qualitative feedback at scale. | |
| Hiring First ML Engineer: Conviction and Transferable Skills | 5 | 3 | 0 | 0 | Brett asks insightful questions about vetting ML talent when non-technical and distinguishing research-oriented from production-oriented engineers. Kevin and Ryan explain transferable ML skills across data domains and reference check signals. | |
| Early ML Architecture: Handling Subjectivity and Real-Time Data | 4 | 5 | 0 | 0 | Kevin details the subjective nature of qualitative coding and the technical tension between batch ML processing and real-time streaming feedback. Ryan introduces the metaphor of first-time versus experienced managers accepting alternative correct answers. | |
| Evolution of Human-in-the-Loop: From MTurk to Yale Faculty | 5 | 5 | 0 | 1 | Brett probes into how Sprig resolved customer expectation discrepancies. Ryan explains moving from Mechanical Turk crowdworkers to hiring a Yale quantitative research faculty member to establish ground truth. | |
| Overcoming AI Skepticism with Mandatory Customer Pilots | 3 | 4 | 0 | 0 | Ryan details the immense skepticism around NLP in 2019 and how Sprig mandated customer pilots on real data with human-in-the-loop validation to achieve a 100% conversion rate. | |
| Evaluating LLMs: Transitioning from BERT to GPT-4 | 4 | 4 | 0 | 0 | Kevin and Ryan explain how they benchmarked Google BERT models against OpenAI GPT models and made the switch once they saw a discontinuous leap in performance. | |
| Mapping Model Capabilities: Question to Study-Level Analysis | 5 | 4 | 0 | 0 | Brett asks how Sprig avoids single-model dependency risk. Ryan and Kevin explain how product scope maps directly to model maturity boundaries, evolving from single-question analysis to study-level synthesis. | |
| Prompt Engineering Breakthroughs in Cross-Question Correlation | 5 | 3 | 0 | 0 | Ryan describes the prompt engineering breakthroughs required to achieve reliable cross-question correlation in survey studies without hallucination. | |
| Reimagining Product Development with AI Engineers Day One | 4 | 4 | 0 | 0 | Ryan outlines why traditional PM/designer duos must now include an AI engineer on day one to validate technical feasibility before designing prototypes. | |
| Pivoting from AI Features to Becoming an AI-First Company | 3 | 4 | 0 | 0 | Kevin and Ryan discuss transitioning from having a single high-effort AI feature to becoming an AI-first organization that provides automated recommendations and fixes. | |
| Preserving User Agency in AI-Assisted Workflows | 6 | 2 | 0 | 1 | Brett offers the Betty Crocker instant cake mix analogy to explain the psychological necessity of user agency in AI workflows. Ryan agrees and details how Sprig positions AI as an assistant rather than a replacement. | |
| Defensibility, Innovation Velocity, and Vision-Based Selling | 4 | 4 | 1 | 1 | Brett asks about the threat of commoditization from foundation models. Ryan rejects the premise that barriers to entry will erase defensibility, citing their proprietary data collection layer and velocity of vision-based selling. | |
| Vertically Integrated AI Squads and Organization-Wide Upskilling | 3 | 4 | 0 | 0 | Kevin and Ryan describe their vertically integrated AI squad structure and internal upskilling initiatives, including company-wide ChatGPT licenses and developer brown-bag sessions. | |
| Strategic Playbook for Founders Navigating the AI Shift | 4 | 5 | 0 | 0 | Ryan lays out three strategic pillars for founders: appointing an AI architect, focusing on real customer problem-solving over tech novelties, and owning the core proprietary data workflow. | |
| Navigating LLM Blindspots and the Horizon of Autonomous Agents | 5 | 5 | 1 | 0 | Kevin points out glaring quantitative and arithmetic weaknesses in LLMs, while Ryan previews autonomous agentic product development workflows inspired by AutoGPT. |
Statements from this episode (25)
Word clouds and topic modeling fail to capture nuanced customer feedback
“You know, people have done word clouds, word counts topic modeling, but, you know, none of those really capture the nuance of what people are saying. And you know, it doesn't account for the fact that people could be saying multiple things per response.”
Machine learning engineering skills transfer broadly across vision, NLP, and audio domains
“One of the things about machine learning and AI in this industry is that there's a lot of crossover between these different data domains. You know, if you're able to solve a computer vision problem pretty well, a lot of those skills are transferable to natural…”
Fast AI evolution requires hiring for learning slope over specific domain experience
“Making sure that the slope is there is really critical as someone who's really smart, just because the AI field is changing and particularly this year, every week, it's rapidly changing. And so you're going to need someone who you can't rely on their prior exp…”
Mandich: User research synthesis lacks an objective universal ground truth
“You can take two expert user researchers, give them the same list of, you know, 500 responses, tell them to distill them down into 10 actionable takeaways, and those 10 will be completely different, or even if they are the same 10 takeaways, the responses that…”
Analyzing machine learning data in batch produces better results than real-time streaming
“From an ML point of view. I think it's advantageous to analyze as much data in batch as possible. You tend to get more information to work with.”
Defining correctness in AI outputs is becoming increasingly subjective
“As we, I think, trend towards artificial general intelligence, the definition of correct or the definition of whether something is, you know, valid or not, It's becoming increasingly subjective”
Three-worker Mechanical Turk consensus often failed expert data accuracy checks
“We originally hired Amazon Mechanical Turks, and we said, hey, we'll save money. We'll have three of them review every response. But we'd often see that even if all three of them gave the same output, we would go to an expert researcher or a customer or look a…”
Sprig hired a Yale faculty member specifically to review model outputs
“We brought on a world-class researcher who was actually on the faculty at Yale as one of our other You know, very early team members. She was actually the fourth person to join SPRIG, and her first role was reviewing.”
AI models will likely never reach total accuracy for subjective feedback categorization
“Cause we will never, it's very unlikely we'll ever get to a hundred percent. And so we'll get, you know, 90, 95, 99. But we'll always make sure that they have the ability to then do that last mile of analysis on their own. And so that's something that we have …”
Tech is likely near the peak of the current generative AI hype cycle
“I think what we're going through now is we're probably close to the peak of the current hype cycle that was introduced earlier this year with, you know, ChatGPT and everything.”
Many AI startups failed between 2018 and 2020 over human-in-the-loop dependencies
“And a lot of startups actually went under in 2018, 2020, just realizing that they were not going to get there.”
A startup's current AI model may be surpassed within three to twelve months
“And so I think for any startup, given the advancements in the LLM field, the current model you're using might be the right model today, but three months from now, 12 months from now, there's a good chance that another model might actually surpass the current m…”
Complex non-deterministic LLM tasks still require manual human evaluation
“And that's just, that's something that's really hard to evaluate in an automated manner, at least right now, because it's a more complex task, because the output is, you know, non-deterministic and freeform. For now, it really just does require some manual eva…”
Much of AI companies' value now lies in prompt engineering
“And I think a lot of the value for AI companies now, you know, is in prompt engineering. I think the early days was actually just Can you build a model? Can you get the model running? Can you get the model working in production? Where I think a lot of it is, c…”
AI product development strictly requires an AI engineer from day one
“The uniqueness that we found for product development specific with AI is that You know, we define product best in class product development as a product manager and a minimum designer starting from the very beginning of ideation and building a product spec and…”
Pitching a product as an automated researcher alienated early customers
“I remember at first the very early days of Sprig, there was some wording around your automated user researcher, you know, or it does all the text analysis for you. And that was actually very off putting for both user researchers and product teams.”
ChatGPT succeeded partly by remaining a standalone product outside core workflows
“And when you look at ChatGPT, I think partly why it's been successful is that it's such a standalone product that's not integrated into anything else. And I think if it actually was integrated into your Slack or into your email or into your work, I think peopl…”
AI product teams must present concrete options rather than asking what to build
“You know, I think with working with customers with AI, you also have to develop world-class techniques of not asking the customer what to build, but instead giving them a set of options to provide feedback on.”
AI engineers must be embedded directly in product squads, not isolated
“And so having, AI engineers embedded with traditional engineers has been the big breakthrough for us and working together To develop and build these features and not seeing AI as, like Kevin said, an input output, you know, the model spits out a response that …”
Hosted LLMs shift hiring demand toward full-stack machine learning engineers
“I think given our current switch to hosted LLMs and away from, you know, taking Google's Burt and fine tuning it ourselves, more towards, I guess we call like a full stack ML engineer, somebody who's able to, you know, help integrate this and actually implemen…”
Many companies will hire AI architects to sit alongside the CTO
“Going forward many companies will bring on AI architects that might even be at the same level or report directly to a CTO to think about what are the models that we're going to use and what are the boundaries of AI and how can we build reproducible AI and thin…”
AI startups lacking workflow or data ownership face imminent disruption
“If you are, you know, a founder or someone working on AI, you really need to think about if you're not owning that core workflow, if you're not owning the data yourself, then it's very easy to be disrupted and someone else could easily, you know, and most like…”
Large language models currently fail at basic math and quantitative reasoning
“Right now a lot of these models fall flat and for an application like ours and for a lot of applications out there, that's kind of a pretty glaring omission. You know, these models are fantastic at summarization. They're really good at generating text based on…”
The next 12 to 18 months of AI will target hallucinations and math
“To your original question, I think issues like those, hallucinations, lack of quant answering, those are probably going to be the major things that get released over the next couple years. I don't know what that looks like. You know, it might be an LLM that's …”
Agentic AI workflows are technically feasible today but face reliability hurdles
“Based on what we've seen today in the boundaries of the LLMs today is that we could, based on what we have tested with the models, actually build that now. And we are seeing the sophistication, the capability of the models to build that now. I think the roadbl…”