Everything Mike Knoop said on any show that made the record, most notable first. Each card names its show and opens the statement there.
Knoop: Scaling Test-Time Compute Will Not Get Us to AGI
“And then there's a new story that's emerged over the last like five months, which is, oh, we're going to scale up this test time compute and that's going to get us to AGI. And I think what V two shows is that that's not quite either. We still need some structu…”
Knoop: Language models operate by memorization rather than solving novel patterns
“Language models. Generally working like a memorization style regime where they're right. Learning lots of data. They're able to apply it to very similar types of patterns that they've seen before, but not novel patterns. That's what RKGI shows.”
Knoop: AGI progress has stalled over the last five years
“You know, my belief is that AGI progress has really stalled out over the last four or five years.”
Knoop: ARC-AGI is the only true AGI evaluation that exists
“Arc AGI to best of my knowledge is the only true. AGI eval that actually exists in the world and measures a actually good definition, correct definition of what AGI is, which we can talk about.”
Knoop: LLMs are high-dimensional memorization, not general intelligence
“Effectively what large language models do today is they are high dimensional memorization systems, right? They are trained on lots of training data. They're able to find and generalize patterns off of the training data that they're trained on and then apply th…”
Knoop: Purely scaling language models will not achieve AGI
“Scaling language models purely will not get there.”
Knoop: Legislating AI research from theoretical capability predictions is dangerous
“I think it's incredibly dangerous to try and make predictions about future capabilities, about where the technology will go and make rules, legislations, laws.
Like prohibiting or enforcing or requiring certain research directions through a theoretical lens.
I…”
Knoop: ARC solution will likely need under 10k code lines, not massive LLMs
“It's quite likely actually that the solution it can be like written in like 10,000 lines of code or less. And it's not gonna require these like, you know, gigantic You know, two hundred billion large parameter models in order to solve it.”
Knoop: ARC Saw No Progress Despite 50,000x Model Scaling
“Surprise that it basically hadn't, and not only hadn't been beaten, there'd basically been no progress in it which I thought was really fascinating given the fact that we've like scaled up these language model systems by almost like 50,000 times over the last,…”
Knoop: Pure LLMs Score 0% and o1 Scores 1% on ARC-AGI-2
“Pure LLM systems are scoring like zero percent now again on on arc B two single COT systems like R one and O one score like one percent.”
Knoop: AI agents will start working in 2025 due to ARC progress
“I actually think we're going to start to see agents start to work this year specifically because of progress on arc.”
Knoop: Startups just doing model training are lighting money on fire
“I think anyone who's like Just doing model training at this point is like lighting money on fire. If you really want to make a unique difference, especially if you're a small startup, like a founder, like you gotta go take an orthogonal approach. You gotta try…”
Knoop: Gary Marcus has been more right than wrong on deep learning
“I generally think he's been more right than wrong. I think if you like just take a limited five year view on this from 20, 20 up until 20, 20, end of 20, 24, you know, I think it was a generally right. Like he was making the right ideas.”
Knoop: Achieving AGI requires merging deep learning with program synthesis
“I actually don't think either is sufficient. I think some merger of the two is what's necessary to get to AGI.”
Knoop: Language models objectively cannot beat the ARC benchmark
“Just sort of objectively, language models do not work to beat Arc. And people have tried.”
Knoop: ARC benchmark solution will likely come from an outsider
“I am more confident actually that or I guess I would bet that the solution arc probably comes from an outsider. I think it's probably gonna come from somebody who's sort of not indoctrinated in the current way of thinking about language models and scale.”
Knoop: Zapier operates with zero internal email across the entire company
“We don't send any internal email.”
Knoop: AI intelligence benchmarks must measure compute efficiency, not brute force
“We do think that efficiency is actually a really, really important aspect of intelligence. You know, you can brute force your way up to intelligence, but we really do want to be shooting for like human targets and efficiency for this stuff.”
Knoop: People should still learn to code for technological leverage
“My hot take is, like, I guess, yes, you should still learn to code. Primarily because it's been, it gives you like leverage over technology today. And yeah, like I don't see that leverage over technology going away anytime soon, particularly if you want to wor…”
Knoop: LLM failure rates break unsupervised server-based automation workflows
“You know, it fails randomly two out of 10 times, which might be fine in a supervised setting like ChatGPT. You know, where you're talking with these sort of assistants but it doesn't really work in an automation case where it's hands-off keyboard running on a …”
Knoop: AI progress remains constrained by fundamental ideas, not compute
“We don't have AGI yet. We don't have the ideas for it yet. We're still our idea constrained even literally today.”
Knoop: Frontier AI labs have stopped publishing technical details
“Frontier AI research is also basically like completely stopped publishing. You know, the GPD four paper had zero technical details. The Gemini paper had zero technical details on the longer context stuff.”
Knoop: ARC-AGI benchmark performance only moved from 20% to 34% in four years
“There's an AI lab called lab 42 out of Switzerland that's been running a small annual contest over the last four years to try and beat this eval and state of the art today is. 34% state of the art four years ago when it was first introduced was 20%. So we've m…”
Knoop: AGI is properly defined as efficient skill acquisition
“Francois definition, which is the one that I think is the right one is this definition that general intelligence is a system that can effectively, efficiently acquire new skill. That's it efficiently acquiring new skill and being able to solve these open-ended…”