The next 12 to 18 months of AI will target hallucinations and math
Kevin Mandich · A guide to building product in a post-LLM world | Ryan Glasgow and Kevin Mandich from Sprig · Sep 7, 2023 · at 1:09:55
Kevin Mandich, Head of ML at Sprig, forecasts the main focus of model advancements over the near-term AI hype cycle.
0:00 / 0:24exact quote · 24.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“To your original question, I think issues like those, hallucinations, lack of quant answering, those are probably going to be the major things that get released over the next couple years. I don't know what that looks like. You know, it might be an LLM that's part of a broader sort of array of models that attack a single input. But since those are the biggest, you know, most glaring Failures of these. I think that's what we're going to see the next 12 to 18 months.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Kevin Mandich
Tech is likely near the peak of the current generative AI hype cycle
“I think what we're going through now is we're probably close to the peak of the current hype cycle that was introduced earlier this year with, you know, ChatGPT and everything.”
Word clouds and topic modeling fail to capture nuanced customer feedback
“You know, people have done word clouds, word counts topic modeling, but, you know, none of those really capture the nuance of what people are saying. And you know, it doesn't account for the fact that people could be saying multiple things per response.”
Machine learning engineering skills transfer broadly across vision, NLP, and audio domains
“One of the things about machine learning and AI in this industry is that there's a lot of crossover between these different data domains. You know, if you're able to solve a computer vision problem pretty well, a lot of those skills are transferable to natural…”
Mandich: User research synthesis lacks an objective universal ground truth
“You can take two expert user researchers, give them the same list of, you know, 500 responses, tell them to distill them down into 10 actionable takeaways, and those 10 will be completely different, or even if they are the same 10 takeaways, the responses that…”
Analyzing machine learning data in batch produces better results than real-time streaming
“From an ML point of view. I think it's advantageous to analyze as much data in batch as possible. You tend to get more information to work with.”
Complex non-deterministic LLM tasks still require manual human evaluation
“And that's just, that's something that's really hard to evaluate in an automated manner, at least right now, because it's a more complex task, because the output is, you know, non-deterministic and freeform. For now, it really just does require some manual eva…”