Stuhlmüller: List-wise re-ranking outperforms per-item scoring in search
Andreas Stuhlmüller · Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit · Apr 11, 2024 · at 48:49
Andreas Stuhlmüller, co-founder of Elicit, explains the technical advantages of using long-context models for ranking search results.
0:00 / 0:28exact quote · 28.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“In the past, I think a lot of ranking was kind of per item ranking where you would score each individual item, maybe using increasingly expensive scoring methods, and then rank based on the scores, but I think list-wise re-ranking where you have a model that can see all the elements is a lot more powerful, because often you can only really tell how good a thing is in comparison to other things, and what things should come first, it really depends on, like, well, what other things are available, maybe you even care about diversity in your results”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andreas Stuhlmüller
Stuhlmüller: AI requires deeper world models to make novel scientific discoveries
“Having deeper models of how, let's see, what are the underlying structures of different domains, how they're related or not related, I think will be an important ingredient for models actually being able to make novel contributions.”
Stuhlmüller: Academia cannot build great software tools due to paper timelines
“It's really hard to actually build interesting tools as an academic. You can't really hire great engineers. Everything is kind of on a paper to paper timeline.”
Stuhlmüller: Generalist AI research platforms will be a winner-take-all market
“So I think there will be, at least within research, I think there will be, like, one best platform, more or less for this type of generalist research. I think there may still be, like, some particular tools, like, for genomics, like, particular types of module…”
Stuhlmüller: Trading inference compute for answer accuracy is undervalued in AI
“Being able to invest more or less compute into getting more or less accurate answers is, I think, one of the core things we care about, and that I think is currently undervalued in the AI space.”
Stuhlmüller: Pure long-context LLMs are significantly harder to debug than RAG
“In one sense, I think you're right that the throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, it's, if things go wrong, it's harder to debug, where, like, if you know, here's the proce…”