Oct 4, 2022 · 53m · knowledge-project
Kenneth Stanley: Set The Right Objectives
0:00 / 0:00
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of The Knowledge Project, artificial intelligence researcher Ken Stanley explains why rigid goal-setting and quantitative metrics hinder breakthrough innovation, advocating instead for the open exploration of counterintuitive stepping stones and subjective curiosity.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Shane holds 16.2% of the talking time here. How this is scored →
Shane as informed peer 4.4 Guest teaching 4.0 Guest disagreement 1.6 Shane pushing back 2.1
speaking balance: gold is Shane, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 36:35 Dismissing speculative visionaries versus stepping stone recognizers
Ken forcefully rejects the conventional lionization of tech visionaries who declare distant objectives like Mars or autonomous driving, calling their claims unfounded speculation rather than genuine vision.
Hardest push from Shane ▶ 38:47 Shane pushes back using Ken's evolutionary frameworkShane intervenes with 'Hold on' and uses Ken's own evolutionary logic against him, arguing that pursuing unrealistic grand objectives still functions as a productive mutation/variation.
Biggest teaching moment ▶ 11:40 Ken explains why metrics blind institutions to real stepping stonesKen breaks down the fundamental deception of complex problems, illustrating that necessary stepping stones are counterintuitive and therefore invisible to naive standardized metrics.
Shane holds their own ▶ 20:01 Shane articulates the algorithmic evolutionary modelShane demonstrates strong conceptual mastery by synthesizing Ken's complex AI research into an evolutionary framework of copying errors, mutations, and node propagation.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Shane as informed peer | Guest teaching | Guest disagreement | Shane pushing back | Why |
|---|---|---|---|---|---|---|
| Modest vs. Ambitious Objectives and Serendipity | 3 | 3 | 1 | 1 | Shane sets up the premise of the book and offers historical examples like rock and roll and penicillin. Ken explains how serendipity operates and draws the distinction between modest and ambitious objectives. | |
| The Paradox of Deception in Complex Goals | 3 | 6 | 2 | 2 | Shane asks how accountability can function in complex systems like education without standardized metrics. Ken delivers an extensive breakdown of the deception problem, explaining why non-intuitive stepping stones render standard short-term metrics counterproductive. | |
| Peer Review and Breaking Monolithic Consensus | 4 | 4 | 2 | 4 | Shane directly challenges whether peer review would naturally suppress counterintuitive ideas. Ken concedes the risk under rigid institutional cultures but argues peer review can decouple individual evaluations from monolithic consensus when structured properly. | |
| Evolutionary Algorithms and Following What Is Interesting | 6 | 3 | 1 | 1 | Shane synthesizes Ken's thesis into an evolutionary algorithm metaphor of mutation, copying errors, and propagation. Ken enthusiastically validates the framing and elaborates on human subjective sensitivity to what is interesting. | |
| Risk Aversion, Subjective Judgment, and Institutional Fear | 6 | 3 | 1 | 1 | Shane introduces the concept of loss aversion and the asymmetric risk profile in public institutions compared to natural selection. Ken agrees, noting metrics act as psychological security blankets that provide an illusion of control. | |
| Visionary Quests vs. Recognizing Stepping Stones | 6 | 5 | 3 | 5 | Shane challenges Ken's critique of bold objectives by invoking Elon Musk's Mars quest and asking if failed grandiose goals are themselves evolutionary variations. Ken differentiates between speculative goal-setting and genuinely recognizing assembled stepping stones. | |
| Institutional Traps, Corporate Innovation, and Startups | 4 | 4 | 2 | 2 | Shane queries the organizational dilemma employees face when pitching non-objective exploration and connects it to startup formation. Ken outlines the structural trap of middle management and bureaucratic grant funding. | |
| Personal Heuristics: Novelty and Avoiding Obvious Ideas | 3 | 4 | 1 | 1 | Shane asks Ken to unpack his personal decision-making heuristics: avoiding ideas that make too much sense and deliberately defying predictable next steps. Ken explains novelty as the primary engine of creative progression. |
Statements from this episode (15)
Stanley: AI research proves objective-driven achievement is deeply flawed
“And in the course of doing that research, we just started to see undeniable evidence that that approach to achievement has some serious flaws.”
Stanley: Major breakthroughs require positioning for serendipity rather than setting objectives
“And actually like a lot of what we do that facilitates making these kinds of important discoveries is to actually set us up, set ourselves up for having effective serendipity, which is not the way that we, you know, talk about things when you talk about settin…”
Stanley: Short-term metric gains in complex problems are deceptive dead ends
“Even if your score on a metric is going up in the short run, it doesn't mean it will all get all the way to the point that you want it to get. That's a fundamental problem. It's called deception. And it's a fundamental problem of all complex problems.”
Stanley: Complex problems like education cannot be solved by direct metric optimization
“We cannot make progress in certain kinds of extremely complex problems, which education is one of those. Simply by just laying out some assessment system and then trying to follow it towards this global objective, which is just incredibly complex to get to.”
Stanley: Metrics are useless for hard problems because required stepping stones are counterintuitive
“If the actual stepping stones that lead to where we want to go are counterintuitive, in other words, they're not what you would expect, then the metrics are useless, right? Because they won't detect those stepping stones because they don't look like what the m…”
Stanley: Complex discovery requires a portfolio of stepping stones, like investing
“Because we don't know the stepping stones, what we need to do is we need to proliferate stepping stones, like stepping stone candidates, things that could lead to something interesting. But we don't know which one. It's a lot, this is related in some ways to i…”
Stanley: Innovation spreads via individual contrarians, not broad consensus
“How innovation actually happens is individual connections. Most people see some idea somebody had and say, well, that doesn't fit with the usual paradigm. Like that's not actually a good idea. Like we know the right way. Now we're all very sure right now what …”
Stanley: Scientific peer review and grant evaluation are flawed by objective thinking
“I think peer review in scientific publication and also in like the assessment of proposals for getting grants, Is, is flawed also because of objective thinking, but it is a place where I think we can if we structure it right, which would be somewhat different …”
Stanley: Evolution solved all problems in one run, unlike typical ML
“Evolution, it's a very unique thing in the sense that it's kind of like a search or like a learning algorithm that discovered everything that was ever created in nature in a single run. This is very different from like what you see in typical machine learning,…”
Stanley: Dismissing experts' subjective sense of 'interesting' cripples innovation
“It's actually the subjective judgments are the interesting ones because, like, the objective judgments are easy. You don't need a degree to just measure something. You know, some kid takes some test, you get a score. You can average them across everybody at th…”
Parrish: Education innovation suffers from small upside and exponential downside risk
“Where we do like we accept failure in science, but we won't accept failure when it comes to an education system. So any, anybody who comes in and tries something else, they have this really weird equation, right? Where you have a very linear, small upside and …”
Stanley: True visionaries spot when stepping stones snap into place
“I think a visionary is somebody, in contrast, who has recognized when the stepping stones actually have snapped into place. Now, that's a person you should follow, and that's a very interesting and unique kind of person. It's a different kind of a person, and …”
Stanley: Direct pursuit of human-level AI is a naive objective
“Like, at some level, there's, like, this really grandiose conception of, like, some human-like computer, and, like, that is, I think, a naive objective right now. Like, you just don't want to know how to do that. We don't know what the stepping stones are that…”
Stanley: Middle managers cannot permit creative exploration because they answer to their own superiors
“The real problem for your boss is they have a boss. How are they going to explain it to their boss? The problem is that this culture percolates through everything. So like everybody's trapped in, even if they believe in what I'm saying”
Stanley: Older luminaries get stuck because deviating from success is scary
“I think this is why like luminaries kind of get stuck in a rut as they get older.
It's part of it.
I think is that it's really,
It's really scary, like, to leave that path, because that's what you're known for, that's what everybody respects you for, and you'r…”