The Exchanges

Q What are the common mistakes that people make in running a premortem?

A One of the mistakes is that they just frame it wrong from the beginning. They'll say, okay, we should do a pre-mortem. So what can go wrong? It's not an issue of what can go wrong. We're dealing with a crystal ball that has showed that the plan has failed. We know that it failed. And our job is, is to, uh, explain why. So say what can go wrong is too tentative. It's too vague. And so that's a, That's I think maybe the most common mistake that, that, that people make is they don't frame it that way. Another mistake is they let, uh, uh, individuals talk too long and, and everybody else is waiting. And this way we're, we're going around the room in a pretty rapid clip and everybody just stating one thing at a time. And a third mistake might be the leader gets cold feet. And says, ah, I just want to hear what everybody else has. And so instead of creating a culture of trust and candor, the leader is, is creating a dangerous environment where people can get in trouble, and the leader is, is being Shielded and protected.

AI assessment note: “One of the mistakes is that they just frame it wrong from the beginning.”

Q One of the things, we're down a few rabbit holes here, and I want to come out of this, but you use the term mental models. What do you mean when you use the term mental model? What is that to you?

A This could be another rabbit hole, but we've, we've, a few years ago, my colleague Joseph Borders and Ron Visayan and I did a study of mental models in petrochemical plants, and going into this study, I thought that a mental model was a set of beliefs about how something works, whether it's a, a, a piece of equipment, a piece of machinery, it's an organization, here, here's how it works, here's, here's what happens, here's the kinds of Of, uh, components and how they fit together. So that was my belief going in, and we spent a week at a petrochemical plant running people through a challenging scenario to, to capture their mental model, and that was part of their mental model, but only part. We found that there was another part of the mental model of not just how does it work, but what are its limitations? What are its boundary conditions? Where can it go wrong? And people who are experienced are well aware of these boundary conditions. And so they, that's part of the richness of their mental model. And then we found another component, which is, um, what are the workarounds that I'm, that I can use if we run into one of those limitations? How can I recover from it? And then we found that there was a fourth component. We have actually a matrix of this, uh, a two by two matrix. The last component Is somebody with a good mental model can also anticipate where people might get confu…

AI assessment note: “now it includes all four of these components, how it works, what its limitations are”

Q I want to come back to something you said about experience. Is there a difference between experience and expertise?

A Right. There, there is a difference. And so one of the things I've examined is how do you know who's an expert? And I identified about seven or eight criteria. And the discouraging part is None of them are foolproof. There is no gold gold standard for who's an expert, but, uh, years of experience certainly contribute to, to expertise, but it isn't the same thing because there are people who just don't reflect on what happened. They don't learn, um, about what happened. Uh, one of the other criteria I use if I'm going to find, uh, gave somebody's expertise is I'll ask them, Tell me the last mistake you made. Let's talk about that. If the person says, I can't think of any mistakes. To me, that means this person may be competent, but it certainly is not an expert. Experts are well aware of their mistakes and their mistakes eat at them until they can, they can sort of imagine what I should have done is this and, and figure out a way around it. So experts are highly aware of mistakes. But people who are journeymen, many of them stay as journeymen because they, uh, they want to move on and forget about their mistakes.

AI assessment note: “There, there is a difference... years of experience certainly contribute to, to expertise, but it isn't the same”

Q together because now you have new information. And we've written a lot at Furnham street on the concept of decision journals. I'm curious as to what information, if you were going to create a journal that an employee had to fill out every time they made a decision, what information would you put in that decision journal to then use at these meetings where you're evaluating their decisions later on?

A I think the idea of a decision journal is a great idea. And I hadn't heard of that before. I would like to know when the employee is making a decision, what is the decision? What are the goals that the employee wants to achieve? The primary goal, but there may be other goals that the employee is aware of. What are the prime pieces of information the employee is using to make the decision? Who are the other people or, or teams that are going to be affected by this decision? Those are the things that I'd like to examine. And then in retrospect, you know, we can see were there, were there, were there cues that I should have been paying attention to that I wasn't? Are there goals that I should have been thinking about that I wasn't? But you have it in the decision journal. You have a record of what was the person thinking about at the time. I think it's a great idea.

AI assessment note: “what is the decision? What are the goals that the employee wants to achieve?”

Q Oh, I don't know if I intentionally make them, but I certainly have more than my statistical fair share of them. What is cognitive flexibility theory?

A Cognitive flexibility theory is the notion of trying to help people achieve expertise by Preventing them from locking in to routines and standard ways of doing things so that they can become more naturally adaptive. You can give them vicarious experiences that can't be handled by the usual routines just to force them out of their comfort zone and to get them into a mindset of not only being prepared to be adaptive, but enjoying being adaptive. Because that's another thing that I've noticed about experts. If they're doing the same job over and over again, they get bored, and then something happens, and they can't use the same routines, and the journeyman, they get really frustrated like that. It's not supposed to happen. I don't know. Who do I call? And the experts, their eyes light up. Ok, we can't use the techniques that have been successful for us up to now. What can we do? What can we, ah, invent? And so the eyes lighting up is like, for me, a key to cognitive flexibility theory is to get people into that mode where they're excited when things don't go as planned, and, and they're gonna have to improvise.

AI assessment note: “Cognitive flexibility theory is the notion of trying to help people achieve expertise”

Q watch in a room with air conditioning and watch this police officer approach a vehicle. It's a whole nother thing for the police officer to be approaching the vehicle. And be, be in a different environment. And that sort of brings me to the question of what role does environment play in our decision-making? And are there things that we can do to improve our environment that improve our decision-making?

A That's a very important observation because this has come up in some of the work that we've done with use of force decisions, where you're, you know, if you're a police officer, you might be in a situation Should I use force or not? And sometimes there's, that's just part of the job. That's a horrible part of the job for police officers. But we find that there may often, there may be sometimes ways to avoid getting into that decision. The way you structure it, the way you frame it, the way you manage it, Can possibly diffuse the situation so that you don't necessarily have to get into a confrontation that you might be able to, uh, to avoid it. So yes, structuring the environment is a much better approach, and this is something we have done a little bit with, not as much as we would like, about, and this, and when I say we, this is my, my friend and colleague John Schmidt. How do you get Of voluntary compliance rather than intimidation. And for many, a number of police officers, they only know one dominant way of getting civilians to comply with them, which is to threaten them, to intimidate them. But we did a project a number of years ago, um, my wife and I, my wife Helen and I did the project to try to see what other techniques For gaining voluntary compliance, for avoiding these kinds of direct confrontations, and the nickname for this project, it was funded by DARPA, Defense…

AI assessment note: “So yes, structuring the environment is a much better approach”

Q individual level, if I'm working in an organization, What can I do within the confines of only what I control? So myself to maximize, um, the insights that I observe and that I can put them into use within that environment. And then I guess the second question would sort of be, what can an organization do to facilitate, uh, people gaining more insight and putting those insights into use?

A You can be deliberate. Like you're describing now, you can say, I want to have more insights, which means taking an insight stance rather than a stance that recoils at something new and unexpected and jarring. You can become curious. You can start to celebrate your own insights, because if you go through the day, we often have very small insights, things we didn't know, and we make these discoveries, and we just sort of, you know, cruise right past them and use them. But if we make a mistake, we beat ourselves up. Right? At least I know I beat myself up. How could I have been so stupid? But I'm less likely to say, gee, that was, that was really clever of me or of people that I'm working with. So you can, you can start attuning yourself to the insights you have and the insights people around you have. So you're more sensitive to that. You can, you can't be curious about everything because then you, you just waste tons of time. But you can be a little curious for a few seconds and say, I wonder what that means, and think about it and imagine it, and just get yourself into that kind of a mindset of wondering and speculating, rather than recoiling at something that's unfamiliar. So that's what you could do as an individual. What can you do as an organization? There, I'm less enthusiastic. If you have an idea that's a bad idea and you make a mistake, that's the down arrow. Everybody…

AI assessment note: “You can be deliberate... What can you do as an organization?”

Q That's a sort of like begs the question, I guess, how do we evaluate people's decision-making?

A How do you evaluate their decision making? You run them through a scenario, see what choices they make, find out what their rationale is, find out the reasons that they're picking one thing over another, and use that to determine if, if, if you think that their mental model is rich enough. Uh, another, I told you, is ask people, tell me, tell me about the last mistake you Or, uh, or you can, you can, you should use years of experience. Just don't take it all that seriously. So there's a variety of criteria that you can use. You can use their track record. Have they had, have they had a track record of successes? But none of these are foolproof. I may have, um, I may be an investor and I have a track record of calling the market right in the last five times. And people say, wow, he really knows what he's talking about. But there are lots of people who are trying to predict which way the market is going to go. And some of them are going to get it right by chance. And people assume it's because they know what they're, what they're doing. And in fact, they're just the lucky ones. So you can't rely on performance. And you can't rely on years of experience.

AI assessment note: “You run them through a scenario, see what choices they make”

Q Can you expand more on what role stories play? Are they just compressions of ideas? Are, are they sort of highlighting certain variables or details or omitting certain variables or details or getting us to anchor to something?

A Stories are doing all of those things, and they're doing it in a, in a way that's, that's really engaging. Like, uh, here's what happened, and I didn't know what to do about it, And then you want to know, what did I do about it? How did it end? And, and, um, I know when, when people have, um, recounted stories in a couple of groups that we studied when we were doing this project, we'd say, okay, so what's the takeaway from the story? And everybody had a different takeaway. So that's the richness of stories is what, what have you learned from the story? What's, what are the implications? And then we, As a small team, just compare notes and realize that because we have different mental models, we have different interpretations of the story and the importance of the story.

AI assessment note: “Stories are doing all of those things, and they're doing it in a”

Q I think that we make, and we don't even recognize that we're making a decision, is who we listen to and who we decide is credible and who we decide is not credible. And we, we don't recognize that we're making a decision in that moment, but how do we sort out who is credible and who's not, or who's, who knows what they're talking about or who's faking it?

A Right. That's, that's a really important question. And I think most people do a mediocre job of determining who they, who could, who they can rely on. And, um, a lot of people go on surface characteristics. So if I carry myself I stand up straighter than usual, and I carry myself with confidence. Uh, people are going to listen to me, and if I express reservations and I use qualifiers, I think this is the case, but I'm not sure. I'm essentially telling people I'm not entirely an expert here, and so I'm, uh, disqualifying myself. I'll give you an example. A friend of mine was excited about a, a, a, a scenario based technique for, for, for training, um, using decision making exercises. And that's a precursor for our, the shadow box method that we're using now. And so he had, um, some, uh, exercises that, that, that he had developed for, he worked in a petrochemical plant. And he had about, oh, 30 years of experience there. And he said, let's, you know, this would be great. Let's use this for, uh, for people who are just have been working outside, checking valves and climbing ladders and things like that. Now they're inside working the panels. So they, they've been promoted. Uh, and there were two people who had just been promoted. And he said, let's get these people up to speed more quickly and let's have them do these These decision-making exercises, and one of the people was som…

AI assessment note: “a lot of people go on surface characteristics. So if I carry myself with confidence”

Q Talk to me a little bit about making decisions in teams of people versus an individual making a decision where boards or committees are, are, are sort of asked to come to a decision. What are the pros and cons of a team-based approach? And what have you seen as more effective ways to get teams to make decisions?

A Team decision making, the teams use different kinds of strategies, and there's different formats. You can have, uh, an autocratic team where a leader just makes a decision. Uh, you can have teams voting. You can have, uh, the vote be, uh, anonymous if people are afraid of repercussions. There are situations like, let's take a pirate ship. In the old Caribbean days when you had pirate ships around, pirate ships were extremely democratic. There was nobody in charge. They voted who was going to lead them. They, they voted about what targets they wanted to attack. But once they were in battle, there was a, a captain. And the captain had, the captain's orders had to be obeyed because under that time pressure with that kind of urgency and those kinds of stakes, You needed to switch your decision strategy to something more autocratic. So there's no general rule for how a team should make decisions. It's going to depend on the situation and the context. I can give you an example of what not to do. There was a movie with Matt Damon a number of years ago called The Martian, and Matt Damon is part of a team that went to Mars And something happened, and they needed to depart, and they gathered everybody together, but they couldn't find Matt Damon. He wasn't available, and it was urgent that they depart, and so they departed without him, assumed that he had gotten injured or killed. That's …

AI assessment note: “there's no general rule for how a team should make decisions. It's going to depend”

Q directly. You approached it from how we can develop expertise to make better decisions. We can sort of reduce errors or have better insight or preferably both. And yet these often seem in conflict with one another. I thought a good place to start this would be What sparks insight and what prevents us from putting our insights into use? Is it because they often contradict the beliefs we hold?

A Uh, an insight definitely is going to be incompatible with beliefs we hold. That's why we're surprised. Uh, I did a study of, uh, a 120 examples of insights. I wonder, I wondered where insights came from. And I did this study out of frustration because I would give talks And I would say, if you want to improve performance, there's two things you can do, and there would be like a down arrow, what you want to reduce, and an up arrow, what you want to increase. And to improve performance, what you want to reduce is errors, and that's what most organizations do. They try to cut down on errors, but they're missing the up arrow of improving, increasing insights. And so most organizations only focus on reducing errors. So I would present this slide And people in the audience would say, yes, that makes sense. That's my organization. All they care about is reducing errors. But then they would ask me the question, so what do you know about insights? And I would answer, I don't know anything about insights. I just have this one slide. I, I haven't studied it. And once I, I gave a print, I used a slide in a talk I gave in Singapore and I got asked that question. And, and then I flew home afterwards, and it's a seven, it was a 17 hour flight. I had a direct flight. That's a long time to be stewing over the fact that I felt like an idiot, not being able to tell them what insights are about. …

AI assessment note: “an insight definitely is going to be incompatible with beliefs we hold.”

Q Is there any sort of practical things that you've seen work particularly well in the wild for people developing mastery in less time, say, than people would expect?

A So let's imagine two people starting out a job. We'll say person A and person worker A and worker B. They're both going to be on the job for a year. Worker A has a lucky year, and everything goes smoothly, and there's no disruptions, no problems. It's just a really very, very, you know, low stress year. Worker B has had a turbulent year. All kinds of things have gone wrong. All kinds of adaptations have been necessary. There were, you know, mistakes made that had to be Identified, diagnosed, and recovered, and one thing is, like, the whole year has been filled with all of this. Who is the one that you would like to have working with you after that first year?

AI assessment note: “Worker B has had a turbulent year. All kinds of things have gone wrong.”