Stuhlmüller: Pure long-context LLMs are significantly harder to debug than RAG
Andreas Stuhlmüller · Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit · Apr 11, 2024 · at 51:48
Andreas Stuhlmüller, co-founder of Elicit, discusses the maintainability trade-offs between RAG pipelines and massive context windows.
0:00 / 0:42exact quote · 42.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In one sense, I think you're right that the throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, it's, if things go wrong, it's harder to debug, where, like, if you know, here's the process that we go through to go from two million papers to an answer, and there are like little steps, and you understand, okay, this is the step that finds the relevant paragraph or whatever it may be. You'll know which step breaks if the answers are bad, whereas if it's just like a new model version came out and now it suddenly doesn't find your needle in a haystack anymore, then you're like, okay, what can you do? You're kind of at a loss.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andreas Stuhlmüller
Stuhlmüller: AI requires deeper world models to make novel scientific discoveries
“Having deeper models of how, let's see, what are the underlying structures of different domains, how they're related or not related, I think will be an important ingredient for models actually being able to make novel contributions.”
Stuhlmüller: Academia cannot build great software tools due to paper timelines
“It's really hard to actually build interesting tools as an academic. You can't really hire great engineers. Everything is kind of on a paper to paper timeline.”
Stuhlmüller: Generalist AI research platforms will be a winner-take-all market
“So I think there will be, at least within research, I think there will be, like, one best platform, more or less for this type of generalist research. I think there may still be, like, some particular tools, like, for genomics, like, particular types of module…”
Stuhlmüller: Trading inference compute for answer accuracy is undervalued in AI
“Being able to invest more or less compute into getting more or less accurate answers is, I think, one of the core things we care about, and that I think is currently undervalued in the AI space.”