The Exchanges

Q So how do you go about doing that? Like the combine also, for those who don't know, I mentioned what the combine is.

A So the NFL combine has almost become its own sporting event. For years it was held in Indianapolis, and they would invite the top, I don't know what it is, 152 hundred draft eligible players. The NFL would invite them. They get up there and they measure them in every possible way. So height and weight and basic vertical jumps and broad jumps, but then also some speed test. The classic one is the 40 yard dash. And then they have position specific drills that they put them through. And While they're doing all of these things, everybody in the league is there. This gathering is, it's a scouts, but then really it is the gathering for the year. So coaches are there, front office folks are there, scouts are there, and everyone's watching these things. They're sitting in the stands, watching these things play out. And the teams then take those data back to their buildings and crunch the numbers and argue what it means. The other thing that happens in Indianapolis is that they interview players. And this is a phenomenal process to watch. They have an informal set of interviews. They just kind of grab some guys and sit them down in the hall and talk to them, but then they have these very formal ones where they schedule them to come through, sit down in the hotel room, and they might have 15 guys in there. They'll have the coaches, they'll have personnel guys, the general manager might b…

AI assessment note: “They get up there and they measure them in every possible way.”

Q And what are the key fields that you've applied it in beyond the research studies?

A The people analytics world really blew up in the last, I don't know, 10 years or so. Google was ground zero and the tech world really kind of bought into this. We need to improve HR decision-making by being more evidence-based. And I was fortunate enough to cross paths with Laszlo Bach pretty early into that effort. And so was spent a fair bit of time in and out of Google over the years. And it was a two-way street, for sure. We learned as much from those guys as they were learning from us. I think they had a great group of folks in there. Financial services was probably the second industry to pick up on that. They really got into it. It's another place where, where talent and human assets are the most important assets, as opposed to a bunch of capital laying around. But the industry that I've really spent the most time with it on is sports. And I had written this paper back in grad school on the NFL draft, and That pulled a little bit of interest out of some NFL teams, and then in the same era that people analytics was blown up, sports analytics was really blown up, and baseball's been doing this for a long time, but then basketball caught fire, and then football is slowly getting there, and I end up working more with organizations and professional sports than any other industry.

AI assessment note: “Financial services was probably the second industry... the industry that I've really spent the most time with it on is sports”

Q Have you tried to study how much you can degrade the model in those instances before you might as well just not have the model in the first place?

A Well, we kind of went the opposite way. We wanted to know how little we could let people play with it and still get their buy-in. And that was what was the most interesting thing about that second paper is that we kind of surprised ourselves by how little we could let them influence the outcome. Let's just play with some numbers. Let's say you give people a chance to use a model. And half the people take it up, and half don't. And the people that take it up easily do better than the ones who don't. So you want people to take it up. And then you let them move the model around by say plus or -10%. The take up jumps up to maybe three quarters. So you get a lot more people using the model, even though they make it a little bit worse. And you think, well, if I got 75% at 10%, I wonder what would happen if I let them do plus or minus five percent. You get the same 75%. You don't get any loss, even though you cut their influence in half. And then you can take it even further. We knocked it down to two percent and we barely lost anybody at all. So it was the act of participating rather than the amount of influence that made the difference, and I think this is very general advice for people who are selling models or analytics. You got to let the users in a little bit. You got to let them kind of put their hands on it, have some influence over what goes into it if you want their full buy…

AI assessment note: “Well, we kind of went the opposite way. We wanted to know how little”

Q All right. Well, this is going to come out a little bit before this year's NFL draft, and I can't have you on without going right to the heart of the matter. So we've got this draft coming up. We've got four quarterbacks looking like they're going to be at the top of the draft. What is interesting to you about what you're seeing happening in this year's draft?

A Well, the consolidation of quarterback talent at the top of the draft is one of the, it seems to me, stories of the last 10 years or so. You see quarterbacks now just drift up the board. A couple years ago, Kyler Murray came out He came out of Oklahoma, and he was a highly rated baseball player, and everyone thought he was going to play baseball, and then he said, ah, maybe I'll play football, and we were speculating. This is like December, January. If he plays football, I wonder how high he'll go. Might he be a first-round pick? No, no way. Sure enough, ok. Then as soon as he announces, very clearly going to be a first-round pick, and then all spring, you just watch him drift up the board until he's the number one pick. One of the reasons for this is I think the league has slowly begun to appreciate position value in the draft It just makes more sense to spend draft capital on positions that are ultimately more valuable, and quarterbacks are the most valuable position in the league. So all teams don't follow this philosophy, and it's been a slow movement, but it seems like slowly the league is learning that the higher value positions, forget the players, just the higher value positions should be drafted more highly, and the best example of that is quarterbacks. And so now this is going to be the most extreme version. We're going to have Four guys in the top 10 or whatever. So …

AI assessment note: “the consolidation of quarterback talent at the top of the draft is one of the”

Q Why don't you take me through your areas of research?

A I started out as a pretty much traditional decision-making guy. I was doing a PhD at the University of Chicago, and they have one of the longest standing decision-making programs there. And then I got pulled a little bit into behavioral economics. That was kind of, in some sense, the dawn of behavioral economics. It was the late nineties. I was there when Richard Thaler moved to the University of Chicago. I had been there one year in an MBA program. And I took his PhD program the first semester he was there. I taught a behavioral economics PhD seminar and a drugger. He allowed me to come in and took a couple of buddies in there with me and kind of got pulled in. He's a decision-making guy as well, but really Had been pushing this behavioral economics frontier, and it kind of took me in the direction of working with data and archival data. It's a little bit different angle for a decision making person, but that's where behavioral economics was going. So that's where I went. And I was always interested in, we call it judgment under uncertainty, but I think normal person would call it forecasting, essentially people's predictions of what's going to happen and what pushes those things around. And within that world, I've been interested in The impact of detecting change in non-stationary environments and how that affects people's judgment and whether people are good at that. And I'v…

AI assessment note: “I've been interested in The impact of detecting change in non-stationary environments”

Q So if you extend that question of that interview to that character assessment, how do you go about putting data in and around making a good forecast on that lens of character?

A The only way to do that is to try to systematize and quantify what is fundamentally a subjective opinion. So a couple of recommendations. One, instead of making a holistic judgment, you need to decompose it. So this is something we know from judgment and decision making that if your judgments are more reliable, if you break them down into the component parts. So instead of saying, well, this guy's got a great character. He's got, or he's got weak character. Let's break it down into work ethic, resilience, coachability, whatever the elements are, you decide what the elements are, but break it down to those elements. And then you've got to turn the subjective thing into something quantified. And so the best practice is to start scoring these things and then track them over time. And it's the same thing we talked about at the top, Ted, about creating data. You, the teams have to create their own data because there are vendors who sell tests and they'll tell you they're good tests. Fine. Collect the data and see whether they actually predict something. But these scouts, one of the main jobs these scouts have is to go out and talk to coaches, Talk to former teammates. Talk to anybody they can who has some window into that player's character in order to get an assessment. So this is an important part of the scouting business, and then the question's whether that judgment can be rende…

AI assessment note: “systematize and quantify what is fundamentally a subjective opinion. So a couple of recommendations.”

Q a portfolio manager, and then you have the analysts who have different thought processes. And there's all the You know, whether you call it politics or compensation arrangements or whatever it is, there are all these reasons why the analyst wants the portfolio manager to like them. How do you kind of balance fostering the independence of thought with a culture that needs to have people communicating with each other?

A For sure. And I recognize that that's a tension and we're not a bunch of little robots that can perform the same regardless of our work conditions and culture makes a huge difference. And frankly, the organizations who have had the best luck with analytics and professional sports have been those who have blended that with culture. It's not analytics or culture is the organizations you can say it's analytics and culture. So The short answer is you can't be pure. You've got to make compromises. But the fact that you have to compromise, Ted, means that you have to push it as much as you can as often as you can. And so you've got to find ways to keep independence where you can, because you know that ultimately you're going to break down the independence. So postponing information sharing as far into the process as possible is something we talk a lot about, because eventually those guys are going to sit around at a table and argue about a player, and so everyone's going to know how everyone else feels about it, but if we're having that kind of information sharing and awareness in March instead of November, we're ahead of the game, because people have been able to independently collect information for a longer period of time. The other bit is that, and I love this example you're giving. I'd love to hear more about it from portfolio managers who do it well. How can they license disagr…

AI assessment note: “postponing information sharing as far into the process as possible is something we talk a lot about”

Q So I think everybody listening probably has some experience with admissions to some school. So I'm so curious, what are the kind of inputs into a model that would determine whether you'd accept a candidate or not to Wharton?

A So mostly, Ted, we haven't changed the inputs. And this is what I think we surprised the administration a little bit. Whenever they sent me to go have these conversations and start talking with these folks. And by the way, the leaders there, in fact, the head of admissions there came out of financial services. And so she was, she was all about this approach in general, which is really helpful. But I think they thought that we'd come up with some new signals, some new silver bullets. We find out, oh, we'll run some numbers and we'll find out when the person says this, or if we see them do that, that's really what tells us. And it's very much not the way we went about it. It was very much You might think of it as process. It's very much the blocking and tackling of decision making. What we did was systematize what they had been doing. All the inputs, especially in the first few years, all of the inputs in the model were the exact same evaluations that they were doing for years and years. It was a reader looking at an application and scoring an essay, or scoring a letter of recommendation, or scoring a transcript. That's all it was. We're taking those inputs, which have been going on for decades, But we're putting them into a model, and we're now going to weigh them systematically. Everybody's going to get the same weight, and we're going to push that against some objectives that …

AI assessment note: “all of the inputs in the model were the exact same evaluations”

Q What's on the leading edge of your research next?

A This question we've been talking about, about individual contributions to teams, I think is the most interesting thing. I'm fascinated by those who contribute to the group outside the box score and whose teams tend to do better. Teams, and I'm speaking very much in, you know, non-sports domains. So the classic example is Shane Battier. So Michael Lewis wrote this article in New York Times Magazine 10 plus years ago now. On Battier's contributions to his teams, despite the title of the article is a no stats all-star. So this guy is an all-star. Eventually he was appreciated, even though he wasn't a big scorer or even a big box score guy at all. So what does that look like in non-sports teams? Are there people out there, and I think we've all experienced some of them, who just make positive contributions, and my projects with them tend to go better than my projects without them. What is it that they're doing? And can we figure out how to identify it and value those people? Because those are the kinds of things that don't typically get promoted. They might not be rewarded at all. They're probably under hired, and yet they're really important. So how can we get at that? And this is back to my little fantasy about meeting analytics. You need a lot of data to get at it, and you need some pretty sophisticated modeling to get at it. But it's one of the questions I'm most interested in,…

AI assessment note: “This question we've been talking about, about individual contributions to teams”

Q So in each of those fields, let's just take it with financial services, the work you've done and the NFL, what ends up being the relevant data?

A So let's start with NFL because there's so much more data, so it's an easier conversation. The big challenge as we move from sports to non-sports organizations is what are the right data? So let's talk about that in one minute. In sports, this is, we're just run over with data. It's been a wonderful thing, and it's blown up even more recently. We've got baseball saver metrics got started with just box score stats, but now we've got motion tracking. It's just a complete revolution in what's available. Teams don't yet use a lot of motion tracking in football. Personnel decisions. Yet. I mean, they're gonna. It's just around the corner. They're beginning to build the base of it. It's the same thing I was just talking about. They don't yet have the history to say, when we see these numbers, it means this down the road. So we're stuck with, well, what does college production mean for NFL production? So how much did the guy play? How did he do? What kind of teams did he play for? And then, of course, the combine. And those are really the two main sources of information. If you're talking about the NFL draft, those are the two main sources of information. You've got production on the field from when the player was in college. You've got some combine numbers, and then you've got anything you can uncover about the player's character. And this is an interesting feature of analytics and p…

AI assessment note: “So let's start with NFL because there's so much more data”

Q So once we have the data set that we want, How do you go about making sure that the process of taking the data and working towards a decision is a good one?

A That's kind of where my work with teams has gone over the years. When I first started going, we had this research on the NFL draft. It says, in short, don't be too sure what you know about these players. And by the way, that suggests that you should probably trade down if you're in one of these top picks. That was kind of our shtick. And then, so we walk in, we have this advice. And that advising may or may not go well, but then over time, if you find the right organization and you build a relationship, it grows into something a little bit bigger than that. And that is, okay, how are you making these decisions anyway? Like what's the process that you assign these scouts and they bring back some data and you aggregate it in some way. And so there's just a lot of steps here. You can imagine some organizations do that well and some organizations don't do it as well. So this is just basic group decision making, and I suspect there's good parallels from the NFL over the asset management, investment, investment teams, or whatever it is. How do you solicit and aggregate a bunch of opinions into an overall decision? We can talk about some good practices and bad practices, but now we're talking about a very general practice. And it has been a fun area to work in because some organizations, even really good ones, Get some basic things wrong. So the simplest one, let's start with this sup…

AI assessment note: “the simplest one... is find ways to keep those opinions independent from one another.”

Q Are there aspects of that call it data-driven scoring system character assessment that you've seen be predictive in the NFL?

A That's really proprietary stuff. So you have to be kind of on the inside. I've only been on the inside of a few teams and the teams I've worked with are always trying to refine that. They're always improving it and they realize it's hard and they realize that they're not there yet. Baseball is probably a little bit ahead. Baseball's been working on this in this kind of systematic way longer than football has, and it's remarkable how similar the two efforts are, and I think the baseball teams are a little bit better about making it systematic. That's even like, I'm talking about some of the best organizations out there, that's one of the main things they're working on. Like, this is one of the frontiers, and it's It's just really hard to, everyone recognizes it's important, but it's so hard that even the best teams and even in the most advanced sports like baseball are still trying to figure out how to do it.

AI assessment note: “That's really proprietary stuff. So you have to be kind of on the inside.”