Apr 11, 2024 · 1h 5m · latent-space
Supervise the Process of AI Research — with Jungwon Byun and Andreas Stuhlmüller of Elicit
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Elicit co-founders Andreas Stuhlmüller and Jungwon Byun join the Latent Space Podcast to discuss building an AI research assistant for scientific literature, detailing their journey from alignment research to developing reproducible notebook workflows, faithful RAG architectures, and scalable reasoning systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 4.8 Guest teaching 5.0 Guest disagreement 1.3 The hosts pushing back 2.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 25:15 Dismissing short-sighted VC skepticism on researcher market size
Andreas directly challenges the narrow VC view that selling to researchers is unviable, arguing that improving broad R&D efficiency unlocks massive economic value.
Hardest push from the hosts ▶ 44:08 Swix challenges model-generated uncertainty calibrationSwix refuses to accept that LLMs can self-report confidence accurately, citing his own empirical experience where models simply hallucinate their calibration.
Biggest teaching moment ▶ 7:25 Jungwon breaks down process supervision for evaluating superhuman systemsJungwon provides a deep explanation of decomposing cognitive tasks into verifiable sub-steps to supervise processes rather than blindly trusting model outputs.
The host holds their own ▶ 52:45 Alessio tests context grounding boundaries with hardware benchmark examplesAlessio demonstrates technical depth by citing an empirical AMD MI300 versus NVIDIA NVLink prompt experiment where document grounding suppressed accurate model knowledge.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Andreas Stuhlmüller's Journey to AI and Founding Ought | 2 | 4 | 1 | 1 | Swix prompts Andreas on his background, prompting an autobiographical overview of his path from early AI exploration to founding Ought. The dynamic is exploratory and narrative with minimal friction. | |
| Jungwon Byun's Background and Ought's Early Research Agenda | 1 | 6 | 1 | 0 | Jungwon delivers a detailed monologue explaining how Ought conducted human-in-the-loop experiments to simulate superintelligent systems before foundation models took off. The hosts listen as she explains process supervision. | |
| Co-Founder Matching and Values Alignment | 4 | 4 | 1 | 2 | Swix and Alessio probe the co-founder matching process and the counterintuitive transition from research alignment to product execution. The guests explain their 50-page evaluation document and mutual alignment. | |
| Defining Elicit: The AI Research Assistant for Systematic Reviews | 4 | 5 | 1 | 2 | Alessio asks for the precise definition of Elicit and questions its scope beyond literature reviews. Jungwon explains how automating systematic reviews and meta-analyses provides the foundation for general scientific reasoning. | |
| Generalist AI Research Platforms vs. Domain-Specific Tools | 5 | 5 | 2 | 3 | Alessio brings up BrightWave to challenge whether domain-specific assistants or a single generalist platform will win. Andreas counters that high-level scientific reasoning is inherently cross-domain and likely a winner-take-all problem. | |
| Market Opportunity, VC Skepticism, and GPT Version Shifts | 5 | 6 | 2 | 4 | Swix pushes on market sizing and investor skepticism that researchers lack money. Andreas and Jungwon defend their thesis by pointing to the massive total economic value of global R&D and foundational truth-seeking. | |
| Tabular Workflows, Concept Grouping, and Building Defensible Moats | 5 | 4 | 1 | 2 | Swix explores Elicit's tabular pivot features and defensibility against new foundational models. Jungwon dismisses generic moat worries by emphasizing that user workflow problem spaces remain vastly larger than current model coverage. | |
| Implementing Constitutional AI for Faithful Abstract Summaries | 5 | 5 | 1 | 1 | Swix inquires about cost management and early Constitutional AI work done with an intern. Andreas explains how they created constitutions and fine-tuned open-source models using RLHF for faithful abstract summarization. | |
| Model Evaluation, Monitoring, and Selecting Model Architectures | 6 | 5 | 2 | 3 | Swix presses on monitoring production hallucination rates and evaluating vision models like Claude Haiku and GPT-4V. Andreas differentiates basic latency DevOps from offline training-time evaluation suites. | |
| Developing Computational Notebooks for Iterative Research Workflows | 6 | 5 | 1 | 2 | Alessio and Andreas discuss transitioning from conversational chatbots to computational notebooks. Andreas explains how notebooks allow defining, debugging, and executing reproducible multi-step data pipelines across thousands of papers. | |
| Human-in-the-Loop Evaluation and Uncertainty Calibration | 6 | 5 | 2 | 4 | Swix expresses skepticism regarding model self-reported uncertainty, stating models often hallucinate calibration. Andreas explains using separate evaluation models (e.g., Llama generating and GPT-4 judging) to achieve reliable uncertainty estimates. | |
| Credit-Based Pricing and Converting Compute to Answer Quality | 5 | 4 | 1 | 1 | The discussion covers credit-based pricing models and variable compute scaling. Andreas explains the philosophy of allowing users to invest arbitrary compute for deep multi-step verification and rigorous answers. | |
| RAG Pipelines vs. Massive Context Windows | 7 | 5 | 2 | 4 | Alessio and Swix challenge the necessity of RAG versus ultra-long context windows. Andreas argues that structured RAG provides essential debuggability and causal attribution that monolithic context windows obscure. | |
| Hard Grounding and Balancing Context with Model Knowledge | 7 | 5 | 2 | 3 | Alessio shares a specific example with hardware specs to illustrate how hard grounding can suppress parametric model knowledge. Andreas distinguishes between strict background-synthesis tasks and truth-seeking agent loops. | |
| Custom Columns and Unexpected Diagnostic Medical Use Cases | 4 | 6 | 0 | 1 | Alessio invites discussion on underrated features, leading Jungwon to showcase customizable extraction prompts and describe an unexpected diagnostic test interpretation workflow in rural clinical medicine. | |
| Systematizing Scientific Discovery and Developing World Models | 5 | 6 | 1 | 2 | Alessio brings up scientific resource allocation and automated discovery. Andreas explains why current models lack the structured world models needed to make surprising, cross-domain scientific discoveries. |
Statements from this episode (27)
Stuhlmüller: Academia cannot build great software tools due to paper timelines
“It's really hard to actually build interesting tools as an academic. You can't really hire great engineers. Everything is kind of on a paper to paper timeline.”
Byun: Supervising step-by-step AI reasoning makes models far easier to evaluate
“The importance of supervising the process of AI systems, not just the outcomes. And so a big part of how, then, like, how Elicit is built is, We're very intentional about not just throwing a ton of data into a model and training it and then saying, cool, here'…”
Stuhlmüller: Elicit co-founders wrote a 50-page mutual evaluation document before starting
“We also did a pretty lengthy mutual evaluation process where we had a Google Doc where we had all kinds of questions for each other, and I think it ended up being around 50 pages or so of, like, various, like, questions and back and forth.”
Stuhlmüller: Elicit builds scaffolding rather than training foundation models
“The way we are building Elicit is not let's train a foundation model to do more stuff. It's like let's build a scaffolding such that we can deploy powerful models to good ends.”
Stuhlmüller: Elicit continues to use T5-based models
“We do also use, like, T-Five-based models, even, even now but started, yeah, started with GPT-II.”
Byun: Scientific meta-analysis typically takes five people over a year
“Lisset was very much inspired by this workflow in literature called systematic reviews or meta-analysis, which is basically the human state of the art for summarizing scientific literature. It typically involves like five people working together for over a yea…”
Byun: Highly structured, reproducible research workflows are uniquely amenable to automation
“Because it's so structured and designed to be reproducible, it's really amenable to automation. So that's kind of the one, the workflow that we want to automate first.”
Stuhlmüller: Generalist AI research platforms will be a winner-take-all market
“So I think there will be, at least within research, I think there will be, like, one best platform, more or less for this type of generalist research. I think there may still be, like, some particular tools, like, for genomics, like, particular types of module…”
Byun: Research, not probability modeling, is the bottleneck in forecasting
“The thing that's blocking people from making interesting predictions about important events in the world is less kind of on the probabilistic side and much more on the research side.”
Stuhlmüller: Seed VCs urged Elicit to build legal AI over research
“We did encounter, I guess talking to VCs for our seed round. A lot of VCs were like, you know, researchers, they don't have any money. Why don't you build a legal assistant?”
Byun: GPT-3 was a qualitative shift, while GPT-4 was an extension
“I think GPT-III was a big change because it kind of said, oh, now is the time to build to you that we can use AI to build these tools. And then GPT-IV was maybe a little bit more of an extension of GPT-III. It felt less like a level, GPT-III over GPT-II was li…”
Byun: Foundational models will not commoditize Elicit due to deep workflow specialization
“I think about this a lot in the context of moats. People are like, oh, what's your moat? What happens if GPT-V comes out? It's like, if GPT-V comes out, there's still like all of this other space that we can go into. And so I think being really obsessed with t…”
Byun: Anthropic's Constitutional AI slashed Elicit's query costs tenfold in days
“At the start of twenty-twenty-three, Anthropik kind of launched their constitutional AI paper and within a few days, I think four days, he had basically implemented that in production, and then we had it in-app, like, a week or so after that, and he has since …”
Byun: Early LLMs prioritized answering questions over faithfulness to source text
“At the time, the models hadn't been trained at all to be faithful to a text. So they were just generating. So then when you ask them a question, they tried too hard to ask, answer the question, and didn't try hard enough to answer the question given the text o…”
Stuhlmüller: Claude Haiku offers an optimal balance of cost and accuracy
“Specifically, I think Cloud Haiku is like a good point on the kind of Pareto frontier, so I think it's like, it's not the, it's neither the cheapest model nor is it the most accurate, most high quality model, but it's just like a really good trade-off between …”
Stuhlmüller: Closed-source models consume most of Elicit's compute budget
“I'd say, like, in terms of number of careers, it's maybe similar. In terms of, like, cost and compute, I think the closed models make, make up more of the budget, since the main cases where you want to use closed models are cases where they're just smarter, wh…”
Stuhlmüller: Notebooks enable debugging and scaling workflows far better than chat
“But the important difference in our minds is with notebooks you can define a process. So in, in data science you can go like, here's like my data analysis process that takes in a CSV and then does some Extraction, and then generates a figure at the end, and yo…”
Stuhlmüller: AI primitives should be semantic tasks, not granular chain-of-thought
“I think chain of thought is maybe still like kind of one level lower on the abstraction hierarchy than we would think of notebooks. I think we'll probably want to think about more semantic pieces, like a building block is more like A paper search, or an extrac…”
Byun: LLM self-reported uncertainty is reasonably well-calibrated in production
“We found it to be pretty calibrated. There varies on the model.”
Stuhlmüller: Separate evaluator models yield better uncertainty estimates than self-evaluation
“I think in some cases we also use the different models for the uncertainty estimates. Yes, then, for the question answering. So, one model would say, here's my chain of thought, here's my answer, and then a different type of model. Let's say the first model is…”
Stuhlmüller: Trading inference compute for answer accuracy is undervalued in AI
“Being able to invest more or less compute into getting more or less accurate answers is, I think, one of the core things we care about, and that I think is currently undervalued in the AI space.”
Stuhlmüller: List-wise re-ranking outperforms per-item scoring in search
“In the past, I think a lot of ranking was kind of per item ranking where you would score each individual item, maybe using increasingly expensive scoring methods, and then rank based on the scores, but I think list-wise re-ranking where you have a model that c…”
Stuhlmüller: Pure long-context LLMs are significantly harder to debug than RAG
“In one sense, I think you're right that the throw everything into the context window thing is easier to maintain because you just can swap out a model. In another sense, it's, if things go wrong, it's harder to debug, where, like, if you know, here's the proce…”
Stuhlmüller: Agentic search ideally balances parametric memory with retrieved context documents
“I think probably the ideal thing looks a bit more like agent control where the model can issue a query that then is intended to surface documents that substantiate its hunch. So I would, that's maybe a reasonable middle ground between model just telling you an…”
Stuhlmüller: In 10-20 years, today's scientific methods will look incredibly unsystematic
“Probably, yeah, I, I'd guess in like, 1020 years, we'll look back and it will be incredible how unsystematic science was back in the day.”
Stuhlmüller: AI requires deeper world models to make novel scientific discoveries
“Having deeper models of how, let's see, what are the underlying structures of different domains, how they're related or not related, I think will be an important ingredient for models actually being able to make novel contributions.”
Stuhlmüller: AI orchestration resembles software engineering far more than ML research
“I think a lot of this looks more like traditional software engineering than it does look like machine learning research, and I think the people who are, like, really good at building good abstractions building applications that can kind of survive even if some…”