Stuhlmüller: Notebooks enable debugging and scaling workflows far better than chat
Andreas Stuhlmüller · Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit · Apr 11, 2024 · at 39:20
Andreas Stuhlmüller, co-founder of Elicit, explains why Elicit moved from conversational chat to computational notebooks for AI research workflows.
0:00 / 0:54exact quote · 54.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“But the important difference in our minds is with notebooks you can define a process. So in, in data science you can go like, here's like my data analysis process that takes in a CSV and then does some Extraction, and then generates a figure at the end, and you can prototype it using a small CSV, and then you can run it over a much larger CSV later. And similarly, the vision for notebooks in our case is to not make it this, like, one-off chat interaction, but to allow you to then say, kind of, if you start and first you're like, okay, let me just analyze a few papers and see, do I get to the correct, like, conclusions for those few papers? Can I then later go back and say, now let me run this over 10,000 papers now that I've debugged the process using a few papers? And that's an interaction that doesn't fit quite as well into the chat framework because that's more for kind of quick back and forth interaction.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Andreas Stuhlmüller
Stuhlmüller: AI requires deeper world models to make novel scientific discoveries
“Having deeper models of how, let's see, what are the underlying structures of different domains, how they're related or not related, I think will be an important ingredient for models actually being able to make novel contributions.”
Stuhlmüller: Academia cannot build great software tools due to paper timelines
“It's really hard to actually build interesting tools as an academic. You can't really hire great engineers. Everything is kind of on a paper to paper timeline.”
Stuhlmüller: Generalist AI research platforms will be a winner-take-all market
“So I think there will be, at least within research, I think there will be, like, one best platform, more or less for this type of generalist research. I think there may still be, like, some particular tools, like, for genomics, like, particular types of module…”
Stuhlmüller: Trading inference compute for answer accuracy is undervalued in AI
“Being able to invest more or less compute into getting more or less accurate answers is, I think, one of the core things we care about, and that I think is currently undervalued in the AI space.”
Stuhlmüller: Pure long-context LLMs are significantly harder to debug than RAG
“In one sense, I think you're right that the throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, it's, if things go wrong, it's harder to debug, where, like, if you know, here's the proce…”