Jun 11, 2024 · 26m · no-priors
No Priors Ep. 68 | With Zapier Co-Founder and Head of AI Mike Knoop
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of No Priors, Zapier co-founder Mike Knoop joins Elad Gil to discuss why LLM scaling alone cannot achieve artificial general intelligence, the motivation behind the million-dollar ARC Prize, and the future of enterprise agentic workflows.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 18.8% of the talking time here. How this is scored →
The hosts as informed peer 5.2 Guest teaching 5.3 Guest disagreement 2.1 The hosts pushing back 1.2
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 7:40 Knoop firmly dismisses LLM scaling as an AGI pathway
Knoop rejects the dominant industry hypothesis that scaling parameters and compute will lead to AGI, asserting that transformer next-token prediction alone will never achieve general intelligence.
Hardest push from the hosts ▶ 5:12 Gil challenges Knoop on LLM generalizabilityGil directly challenges Knoop's skepticism of LLMs by arguing that foundation models exhibit unprecedented cross-domain generalizability and unlock economic value across diverse subtasks.
Biggest teaching moment ▶ 3:38 Knoop reframes AGI measurement around sample efficiencyKnoop breaks down the flaw in consensus economic definitions of AGI, contrasting narrow AI task mastery with human-level, sample-efficient acquisition of entirely novel skills.
The host holds their own ▶ 23:38 Gil contextualizes open source via 1990s crypto regulationsGil demonstrates deep historical expertise by synthesizing how open network protocols and 1990s cryptography regulation battles mirror current attempts to restrict open-weight AI development.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Introducing Mike Knoop and the ARC Prize Launch | 4 | 5 | 2 | 0 | Gil opens with standard framing introducing Knoop and the ARC Prize. Knoop delivers an extended monologue critiquing industry consensus, arguing that LLM scaling has stalled true AGI progress and that frontier labs have ceased publishing meaningful details. | |
| Defining AGI: Economic Output vs Skill Acquisition | 4 | 6 | 3 | 0 | Gil invites Knoop to contrast the consensus definition of AGI against his own. Knoop dismisses the economic output metric as reflecting routine human labor rather than general intelligence, educating the audience on Francois Chollet's sample-efficient skill acquisition benchmark. | |
| Why LLMs and Pure Scale Fail at AGI | 6 | 6 | 4 | 5 | Gil pushes back twice, arguing that LLMs exhibit broad generalizability across subtasks and asking whether scale plus reasoning modules could bridge the gap. Knoop firmly counters that LLMs are merely high-dimensional memorization systems that inherently fail on novel out-of-distribution reasoning. | |
| Exploring Program Synthesis and Neural Architecture Search | 4 | 6 | 1 | 0 | Gil asks which alternate technical paradigms show promise. Knoop outlines program synthesis and unconstrained neural architecture search powered by modern cheap compute, explaining how they circumvent human inductive bias. | |
| Assessing AI Risk, Sentience, and Empirical Governance | 7 | 5 | 3 | 4 | Gil brings up Peter Watts' Blindsight to interrogate the distinction between intelligence and sentience, probing the existential risk implications of accelerating AGI. Knoop sidesteps metaphysical debates, advocating for an empirical governance framework over theoretical preemptive restrictions. | |
| Mobilizing Outsider Researchers Through the ARC Prize | 4 | 5 | 2 | 0 | Gil asks why Knoop structured the initiative as a prize competition rather than venture funding. Knoop explains that breakthroughs are more likely to originate from unindoctrinated outsiders creating lightweight algorithms under 10,000 lines of code. | |
| Zapier's AI Strategy and Natural Language Automation | 5 | 5 | 0 | 0 | Gil asks about Zapier's internal AI adoption journey and concrete usage metrics. Knoop details their early R&D on custom agents and shares metrics of over 50 million executed AI tasks along with the rollout of natural language bot configuration. | |
| Expanding Agentic Workflows with Behavioral Guardrails | 6 | 5 | 1 | 0 | Gil asks about the trajectory towards autonomous agentic workflows. Knoop breaks down his framework of concentric circles of risk tolerance, explaining why enterprise adoption requires granular clamping mechanisms on agent behavior. | |
| Defending Open Source Research and Pragmatic Regulation | 7 | 5 | 3 | 2 | Gil draws parallels to 1990s cryptography export regulations and open internet protocols to discuss open-source AI policy. Knoop strongly defends open-source research as vital to preventing stagnation and rejects theoretical regulations that curtail independent research. |
Statements from this episode (15)
Knoop: AGI progress has stalled over the last five years
“You know, my belief is that AGI progress has really stalled out over the last four or five years.”
Knoop: Frontier AI labs have stopped publishing technical details
“Frontier AI research is also basically like completely stopped publishing. You know, the GPD four paper had zero technical details. The Gemini paper had zero technical details on the longer context stuff.”
Knoop: ARC-AGI is the only true AGI evaluation that exists
“Arc AGI to best of my knowledge is the only true. AGI eval that actually exists in the world and measures a actually good definition, correct definition of what AGI is, which we can talk about.”
Knoop: ARC-AGI benchmark performance only moved from 20% to 34% in four years
“There's an AI lab called lab 42 out of Switzerland that's been running a small annual contest over the last four years to try and beat this eval and state of the art today is. 34% state of the art four years ago when it was first introduced was 20%. So we've m…”
Knoop: AGI is properly defined as efficient skill acquisition
“Francois definition, which is the one that I think is the right one is this definition that general intelligence is a system that can effectively, efficiently acquire new skill. That's it efficiently acquiring new skill and being able to solve these open-ended…”
Knoop: LLMs are high-dimensional memorization, not general intelligence
“Effectively what large language models do today is they are high dimensional memorization systems, right? They are trained on lots of training data. They're able to find and generalize patterns off of the training data that they're trained on and then apply th…”
Knoop: Language models objectively cannot beat the ARC benchmark
“Just sort of objectively, language models do not work to beat Arc. And people have tried.”
Knoop: Purely scaling language models will not achieve AGI
“Scaling language models purely will not get there.”
Knoop: Cheaper compute justifies revisiting unbiased neural architecture search
“And I suspect now over the last four years, we have. We might now have enough compute that's come online at a cheap enough, like kind of cost per flap that some of those old neural architecture search methods we should revisit and relax the search.”
Knoop: Legislating AI research from theoretical capability predictions is dangerous
“I think it's incredibly dangerous to try and make predictions about future capabilities, about where the technology will go and make rules, legislations, laws.
Like prohibiting or enforcing or requiring certain research directions through a theoretical lens.
I…”
Knoop: ARC benchmark solution will likely come from an outsider
“I am more confident actually that or I guess I would bet that the solution arc probably comes from an outsider. I think it's probably gonna come from somebody who's sort of not indoctrinated in the current way of thinking about language models and scale.”
Knoop: ARC solution will likely need under 10k code lines, not massive LLMs
“It's quite likely actually that the solution it can be like written in like 10,000 lines of code or less. And it's not gonna require these like, you know, gigantic You know, two hundred billion large parameter models in order to solve it.”
Knoop: Over 50M AI tasks have run on Zapier
“Over fifty million AI tasks have run on the platform to date over the last year and a half or so since we started tracking.”
Knoop: In-workflow AI is the dominant adoption pattern today
“Using AI in the middle of a workflow is, is kind of the dominant way people are adopting AI today.”
Knoop: Narrow AI should be regulated through existing frameworks
“I mean, I think my sort of underlying beliefs on AI are AI should likely get regulated through the existing regulatory frameworks that exist. I don't see a lot of new harm or use cases or damage caused by just the narrow form of AI systems that we have today t…”