The Exchanges

Q see which of those experiments leads to some interesting insights or conclusions. And then the idea being we take those conclusions Uh, or that interesting insight. And then we propagate it to all the other nodes, almost like, uh, nature sort of rewarding variations. And we do this blindly because we, we don't know sort of what will yield the best results. Is that, how do you think of it?

A Yeah, that's a good characterization. Actually, it's not a, it's not a coincidence that you bring up evolution because the, the background behind the genesis of this discussion in the book is that, you know, I was working in evolutionary algorithms. And this is like in a branch of artificial intelligence. And I was trying to understand what actually allows evolution In nature to make the kinds of incredible innovations that it makes. You know, if you think about it, like from a computer science perspective, like as opposed to biology perspective, like what evolution is, it's a very unique thing in the sense that it's kind of like a search or like a learning algorithm that discovered everything that was ever created in nature in a single run. This is very different from like what you see in typical machine learning, which is like, We're going to try to solve a very hard problem and just that one problem and all the resources go to that one problem and that's the objective and that's the run. It's a very, very unique that you would have a single run, discover the solution to every problem. Um, and so what I mean by problems, like how do you get flight to work? How do you get photosynthesis to work? How do you get human level intelligence? Like it's all one run. Um, and I was trying to understand this from an algorithmic perspective, not like a biology textbook, but like, can we a…

AI assessment note: “Yeah, that's a good characterization. Actually, it's not a, it's not a coincidence”

Q You had two sort of insights on decision making that you mentioned before we, we got on here. And the first was that you told me one rule of thumb that you use for deciding on projects is to try to avoid ideas that make too much sense. Can you double click on that?

A Yeah, this is, it's kind of fun because when I say it, people look at me like I'm insane. It's like somebody says, look at this great idea. Like it's usually something in science. It's like, uh, this is Isn't this, isn't this exciting? Or maybe even we should follow up on this or do something. And I'm just like, well, I do think it's a, it's a good idea, but it makes too much sense. So I really don't find it that exciting. And so that's a little personal heuristic, which is related to what we're discussing. Um, and it's sort of what it is for me or a rule of thumb is that like, I recognize because of all this, that when you talk about stepping stones that lead somewhere Really revolutionary. They're going to be counterintuitive, you know, and if you think about it, that makes a lot of sense because if they were just intuitive, like obvious, then we would have crossed them anyway, you know, like it's not a problem. Getting to things that are really important or interesting will cross through counterintuitive stepping stones. And so what that means is that like, they won't make sense at first. That's basically the definition of counterintuitive. Like it's true that in hindsight, they'll make sense. Cause like at some point you look back and you say, oh, well, I see why this led to that, but looking forward, they will be strange and counterintuitive and basically seem like they do…

AI assessment note: “stepping stones that lead somewhere Really revolutionary. They're going to be counterintuitive”

Q What do you mean when you say novel?

A Yeah. Novel means it hasn't been tried before. It doesn't look like things that have come before. Um, it's different in some, uh, very fundamental way from other things, um, that have existed in the past. Um, and, and so those are like, like think about things like, you know, what's, what's novel changes over time, of course, like something like, you know, the idea of like a little room that's like on top of wheels and can move you from A to B, like, you know, a hundred years ago, a 120 years ago, like that would be like a super interesting thing. Like if you actually could get something like That would be, like, dinner conversation. Could you imagine? How will this affect society? That, like, this little thing on wheels can, like, go anywhere. Now, today, that's not very novel. Like, it's completely not novel, and that's why it's not interesting. Like, this is not good dinner conversation. Like, because it's been done, and it's been done for a hundred years. And so, like, it changes. And so, generally speaking, like, the next stepping stones, the ones that are going to be interesting, are also going to be novel. So that's generally a rule of thumb. And so like running away, like novelty means running away from where you've been in the past. Um, you know, a lot of objective problem solving is the opposite. It's about converging along the same path. Convergence is basically what…

AI assessment note: “Novel means it hasn't been tried before. It doesn't look like things”

Q this at any of those levels and we didn't because that would require a, it would require sort of like trying something that might fail, which I want to talk about in a second, but B it would require us Uh, saying something subjective, that this is a better teacher than somebody else, and we're so hesitant to do that. Can you, what are your thoughts when I say that?

A Yeah, I, I agree with the point. Like we could, we could debate about whether we should have, um, one teacher, like teaching everybody in New York state or something like that. I mean, I'm not saying we shouldn't, but we could debate about it, obviously. But the, but the point that you're making that like, we can't even like consider it because of the structure of the system, that is a problem. Why can't we debate about it? Like, in an official sense. I mean, we could, you and I could debate about it, and it will have no effect on anything, but, but, like, the actual official system, the formal system that actually has the gatekeepers inside of it will not ever go through that debate. Um, because, yeah, it's like, it's incompatible with the assessment system, and it's just, like, way too unwieldy, given that bureaucratic system, the way it's set up. And I think there, that's an example of what is impeding our ability to collect stepping stones and try different things and see their outcomes and Really be flexible and innovative and sort of like proliferate ideas in the spirit of nature. Um, but it's, it's, I think it's multifaceted. So it's not only, you can't point to just one reason for it. It's like pervasive across all kinds of levels of the culture that we're, we're afraid of this kind of engagement and exploration, including what you alluded to briefly, just that we are a…

AI assessment note: “Yeah, I, I agree with the point.”

Q But, but hold on, like at a meta level, and it might be completely wrong here. Isn't that a variation? Like we're trying this idea now and it doesn't work. And just because it doesn't work doesn't mean that we shouldn't try it because That in and of itself is a variation that we're, we're, you know, this is like how nature proceeds.

A So, yeah, I think this gets into some subtlety. Um, like, is it ultimately, um, you know, actually harmful to just try something, um, even if like, it's really unrealistic. Um, and like it, I think it depends on what your motivations, like if you're going in that direction, cause it's interesting, it might not be harmful. Um, I think the field of AI is kind of like that. Like, at some level, there's, like, this really grandiose conception of, like, some human-like computer, and, and, like, that is, I think, a naive objective right now. Like, you just don't want to know how to do that. We don't know what the stepping stones are that lead to that, although they're getting closer, perhaps, but they're still not close, I would say. Um, but at the same time, like, the, the point that, like, investigating Around the area of, like, algorithms that have intelligent, like, qualities. That's still valuable, I think, because it's interesting. Like, one thing that happens, I think, if you do it, is you're, you're unearthing stepping stones that could lead to something, um, else that isn't artificial general intelligence, but still really valuable. That's not usually how it's thought of. I guess in some way that's going to be disappointing if it's like, well, I, Made some progress, but it's never going to get to a, to the AI that I'm envisioning, but it still caused something cool to happen…

AI assessment note: “I think it depends on what your motivations, like if you're going in that direction”

Q why you can't be so tied to your destination that you're not open to the unexpected and unplanned, and why you should avoid ideas that make too much sense. It's time to listen and learn. You wrote a book about questioning the value of objectives, which revealed a surprising paradox that objectives are good when they're modest, but things get more complicated when they're ambitious. Can you expand on that?

A Yeah, this is something that people don't talk about that often. Um, people don't talk about the problem with setting objectives. Um, there, there are some controversies in society. This is currently not one of them. Um, but yet if you think about it, this is something that we do all the time. Basically one of the, like, I think deepest facets of our culture is that we think of accomplishment and achievement and discovery in terms of Setting an objective and then pursuing it. And I, I do research in artificial intelligence, my normal job. And in the course of doing that research, we just started to see undeniable evidence that that approach to achievement has some serious flaws. And that realization at first was mostly just an kind of algorithmic realization, like, okay, well, this has, this has applications and implications for artificial intelligence. But it dawned on us over time that actually it has a lot more implications than just for artificial intelligence, because it's not just something people do in the algorithms of artificial intelligence, but in basically in life and in our culture, it's what we do all the time. And it started to seem to me maybe almost urgent that this is actually brought into like a public conversation of some sort. So that's why myself and my coauthor, uh, Joel Lehman decided to To kind of take the unconventional road of writing a book like this…

AI assessment note: “we just started to see undeniable evidence that that approach to achievement has some serious flaws”