The Exchanges

Q There are a bunch of other techniques that we see investment teams use to try to mitigate risks upfront. What are some of the, your favorite ones in addition to premortems or, well, some of the ones you don't like as much?

A Let me start with the ones I don't like that much. So one of them is the devil's advocate approach, which gets a lot of publicity. I'm only aware of one study that tried to evaluate the effect of a devil's advocate to see if it had an effect, and they got a significant result. It was significantly worse than not using a devil's advocate. And the reason seems to be, if we're a team and I appoint somebody to be the devil's advocate, then I don't know if the person is coming up with these ideas because the They're of genuine concern, or because that's the person's role, but I assume it's because it's their role, and they don't really necessarily believe in it, so it's easy to discount a devil's advocate. It also means I don't have to be critical. I've outsourced it to a devil's advocate, so that's not healthy for a group.

AI assessment note: “Let me start with the ones I don't like that much. So one of them”

Q I know Paul and Paul have been on the show before, so I think it'd be fun given that they were bringing you to the table for this paper we're going to talk about. Why don't you go through your background and areas of interest?

A Sure. So, I'm a cognitive psychologist. My primary area of interest is expertise. How do people become highly skilled? And that started in the mid-seventies, and when all of a sudden the Arab oil embargo hit, and you had all these pilots. I was working as a research psychologist for the Air Force at the time. All these pilots used to fly as much as they wanted. Now they couldn't, because jet fuel was too expensive. And so they had to learn in simulators, which they always had considered as toys, but now they couldn't anymore. And so my job was to figure out what is expertise, and how do you develop it, and how do you develop it most quickly.

AI assessment note: “I'm a cognitive psychologist. My primary area of interest is expertise.”

Q Now, if you were conducting a premortem on the premortem process, What are the things that will commonly go wrong that makes a plan of a premortem fail?

A Well, that's one of the reasons that I've been writing this piece with Paul Johnson and Paul Solkin. We see people making mistakes. There are ways that the premortem will fail. First of all, if the leader doesn't have buy-in, then the leader will come up with something trivial, and everybody knows they have to not surface their prime concerns and choose something that's safer and minor. So that's one part. We see premortems fail when the person who's facilitating will say something like, okay, we've seen the plan, let's do a premortem, what could go wrong with the plan? That's the wrong approach. What can go wrong with the plan? That's a typical critiquing approach. You can't say what could go wrong. You say, I'm looking at a crystal ball. The plan has failed. There's no doubt. That changes the mindset that instead of just sort of musing about it, now you're seeing a disaster, and you're now in a mode of what caused that disaster. It changes the kinds of issues people identify.

AI assessment note: “There are ways that the premortem will fail. First of all, if the leader”

Q If we go back then to you're doing this research to try to distill, you know, how to help these fighter pilots and simulators, and then you learn that, well, the real way that they do it, the real way they develop expertise is in their heads. What do you do with that as a conclusion to research?

A Well, there was no conclusion for the work with the fighter pilots because we were building simulators, so we just had to make the simulators as effective as possible. Now, the fighter pilots and the whole community had the belief you had to make the simulator look as much as possible like the real thing. That was wrong. It doesn't have to look like the real thing, and you don't have to waste a lot of money on that. You have to have really good training scenarios that put people through their paces. And so that was the big aha from that project. But where the expertise work came in was much later when I started studying decision making. And up to that point, there was a belief that a rational decision maker lays out all the options, lays out all the evaluation dimensions, rates each option on each dimension and so forth. When's the last time you ever, I mean, people sometimes do it, but it's very rare and it's not an adaptive strategy. But that's what the field thought was the gold standard, because the field had mostly studied college sophomores handling unfamiliar tasks in laboratory environments, and so the field hadn't studied experts, and so because of my interest in expertise, I was able to look at decision making and say, where's the expertise here, and that was the formulation for my recognition prime decision model, and the discovery we made is that people don't Genera…

AI assessment note: “where the expertise work came in was much later when I started studying decision making”

Q Once you identified that the recognition strategies are the way that people go about making decisions, How did you then study different ways that people can improve their decision processes?

A That's where it really gets important, isn't it? What can you do about it? And so for maybe 10 or 20 years, we were looking for how do we turn this into training and into preparation so that we can help people become better decision makers. And we came up with nothing. And so it was very discouraging. And then about six or seven years ago, I was exposed to a technique that we call the shadow box. Which is a technique that we've been using it for the last five years, and it wasn't developed by me or my team, developed by a guy named Neil Heinz, who's a retired New York Fire Department battalion commander, and he had developed this in the wake of the nine 11 attacks, and how do people handle situations that no training could have prepared them for, because nobody could foresee them, and the way Shadowbox works, it's a scenario-based approach, So I'd give you a scenario, and we'd walk through the scenario, and then at a certain point, I'd stop, and I'd say, ok, Ted, here's four options of what you can do at this point. Rank order them, and write down the rationale why you rank ordered them. But it's not just about the options. We may go a little bit further, and then I'd stop, and I'd say, ok, Ted, here's three goals that you could be pursuing. Rank order them in terms of what's the most important to achieve to the least, and write down your reason. Then we might go a little bit f…

AI assessment note: “I was exposed to a technique that we call the shadow box.”

Q As you discussed that and looked at different applications, were there areas where it was highly effective and others where it's less effective?

A We haven't found many areas where it's not effective, and we looked. We said, we're not teaching this strategy. This strategy describes what people naturally do. So we said, maybe it doesn't get applied when there's less time pressure. So we looked at non-time pressured situations. We looked at designers, and they still were predominantly using a recognitional strategy. And then we said, maybe if you don't have much experience to begin with, it's not a strategy you'll use. So we We looked at novices. We looked at soldiers at Fort Knox, tank commanders who hadn't really had much experience in tanks, and we said, what about them? They were the group we've seen that did not use recognition decision-making. They did look at options. They tried to identify the best option from a set. It didn't help them. They still did a terrible job, but they had no basis and expertise to use recognition strategies.

AI assessment note: “We haven't found many areas where it's not effective, and we looked.”

Q We hear a lot about checklists and Atul Gawande's checklist manifestos. How do you blend the necessary tasks with the intuition that comes on top of that?

A Right, and that's an important part, is that it is a blend, and Gawande, you know, comes up with these great checklists, but in the book, he describes how much work it is to come up with a reasonable checklist, and you can't have too many items, you can't make it too complicated, and so there's a real art to coming up with a checklist. How do you blend the other part? You break people out of the mindset. My job is simply to learn the rules and learn the procedures, and a lot of people do have that belief. And you've got to get them out of that mindset, and get them into accepting that life is too complicated, and they are going to be encountering situations that none of the checklists have prepared them for, and they better be adaptive and resilient in order to handle them.

AI assessment note: “You break people out of the mindset. My job is simply to learn the rules”

Q And what did you find in that initial research?

A So I'll tell you a story. I was doing a project and was down at Tyndall Air Force Base, and there were these two fighter pilots, F-. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . And takes a task, break it down into the components, and that's the way you teach it. And I said, ok. And they said, we're the ISD specialists for our group. And I said, ok, that's nice. And we're building the IOD specs for how to perform a lot of our flying tasks. And I said, ok, so what's the problem? And they looked around to make sure nobody was overhearing them, and they said, well, we've written all the specs, but we don't follow any of those rules. That's not the way we fly. So I said, okay, just write down the rules you follow, and they said, we don't follow any rules. We just know what to do, and that's what I've been hearing time and time again from firefighters and others, that there's a belief that you just follow rules and procedures. For complex tasks and ambiguous situations with lots of uncertainty, there's no set of rules that are sufficient, and an expertise is It's not about just incorporating all the rules. Expertise is about building tacit knowledge, and that's the core. That's the primary thing that I've learned is how to go after tacit knowledge.

AI assessment note: “Expertise is about building tacit knowledge, and that's the core.”