The Exchanges

Q whether, whether the margin of the business can be maintained at the current prices. And I thought, okay, well, forget about it because these labs have such an economically valuable tool for us that they'll raise prices. But we might be in the moment where they're going to get into a price war because OpenAI is rumored to be potentially dropping prices. And so what do you think about that?

A I think we're absolutely right on the precipice of that. I think you actually saw in the audience here by comparison, we saw every hand go up when you said, would you be willing to pay double when we were together three months ago in April? And when, when Alex asked if people were willing to pay four and five times as much, there was still a quarter of the hands in the room that were up. And we're talking a room of about 200 people roughly, right? So it was a lot of people was a good, you know, good, good tea sample, so to say. I contrast that to what we just saw right now, and I think it's a fairly, you know, even distribution, similar subset of people. And the reality is there's there's more skepticism of value that they're getting from it, especially when you start layering on the access and the capabilities associated with Some of the models that are still per seat, as well as some of the open models, which you can get access to and, you know, for free, you can do a lot of really cool things.

AI assessment note: “I think we're absolutely right on the precipice of that.”

Q to see this continue, agents can't be limited. They have to be able to operate autonomously and spend all those tokens and be effective. So what are the limits that you're seeing with agents today? And do you think that the, like, if we could extrapolate a little bit, it means that we're going to see some more speed bumps as the labs try to roll this technology out further?

A Yeah, I mean, you use the term limit. I think it's, um, I think it's, uh, a function of both risk tolerance, um, as well as cost tolerances. And then finally, like, what is it you have an expectations of these things doing on their own? And so the limitations are actually in all in all three of those areas that are coming through. Um, you know, from a risk tolerance point of view, I think people are saying, wait a second, I'm worried that the agent without some level of, you know, Call it control or governance around it. It could go just about anywhere. And what does that mean within my organization, depending on what access I give to it from a data perspective? Um, you know, I would say from client data or whether it's, you know, it's code itself and what can it do to change code? If you ask to do one thing in one area, will it simply think that it needs to do that everywhere else? And there's a, you know, we'll say the ability to extrapolate on a single point and like what control exists there. So that's the first limit. The second limit is on, like I said, on the cost experiments, variance. We went through there a second ago, which is, Hey, look, like there are just things you're not going to want it to do because back to the MIT study, there might be things that humans can do not only better, but more cheaply, especially now, depending on, you know, if you're, if you have a…

AI assessment note: “a function of both risk tolerance, um, as well as cost tolerances.”

Q whether, whether the margin of the business can be maintained at the current prices. And I thought, OK, well, forget about it because these labs have such an economically valuable tool for us that they'll raise prices. But we might be in the moment where they're going to get into a price war because OpenAI is rumored to be potentially dropping prices. And so what do you think about that?

A I think we're absolutely right on the precipice of that. I think you actually saw in the audience here by comparison, we saw every hand go up when you said would you be willing to pay double when we were together three months ago in April. And when, when Alex asks if people were willing to pay four and five times as much, there was still a quarter of the hands in the room that were up. And we're talking a room of about 200 people roughly, right? So it was a lot of people was a good, you know, good, good tea sample, so to say. I contrast that to what we just saw right now, and I think it's a fairly, you know, even distribution, similar subset of people. And the reality is there's there's more skepticism of value that they're getting from it, especially when you start layering on the access and the capabilities associated with Some of the models that are still per seat, as well as some of the open models, which you can get access to and, you know, for free, you can do a lot of really cool things.

AI assessment note: “I think we're absolutely right on the precipice of that.”

Q to see this continue, agents can't be limited. They have to be able to operate autonomously and spend all those tokens and be effective. So what are the limits that you're seeing with agents today? And do you think that the like, if we could extrapolate a little bit, it means that we're going to see some more speed bumps as the labs try to roll this technology out further?

A Yeah, I mean, you use the term limit. I think it's, um, I think it's, uh, a function of both risk tolerance, um, as well as cost tolerances. And then finally, like, what is it you have an expectations of these things doing on their own? And so the limitations are actually in all in all three of those areas that are coming through. Um, you know, from a risk tolerance point of view, I think people are saying, wait a second, I'm worried that the agent without some level of, you know, Call it control or governance around it. It could go just about anywhere. And what does that mean within my organization, depending on what access I give to it from a data perspective? Um, you know, I would say from client data or whether it's, you know, it's code itself and what can it do to change code? If you ask to do one thing in one area, will it simply think that it needs to do that everywhere else? And there's a, you know, we'll say the ability to extrapolate on a single point and like what control exists there. So that's the first limit. The second limit is on, like I said, on the cost experiments, variance. We went through there a second ago, which is, hey, look, like there are just things you're not going to want it to do because back to the MIT study, there might be things that humans can do not only better, but more cheaply, especially now, depending on, you know, if you're, if you have a…

AI assessment note: “the limitations are actually in all in all three of those areas”

Q to token maxing. It's called token minimizing. And you're seeing companies like AT&T and Meta get really serious about cost. And we also know that Uber, of course, spent their entire budget in less than half a year. Who do you think is going to win out at the end of the day, the token minimizers or the token maxers, given the definition of token maxing you just gave us?

A Yeah, I mean, I think here's the good news. The good news is I don't I don't think there's a winner and loser That's going to be defined by did you token max or not token max? I think the winner and loser is going to be defined by did you outcome max or not? It's in part going to be a function of how did you incentivize people, which goes into that leaderboard and the things that got a lot of folks will say in trouble or certainly in the news, right? So there's that piece of it. It's the how do you actually want to encourage people to do it without encouraging the wrong things? It's take it too far, et cetera, right? So there's that piece or spend too much money. And then the other part of it is going to be actually, I think, from a planning point of view within an organization, and I deal with this as well. So I sit on the boards for for our US and our global organization at PwC, and we talk about this a lot. In fact, we talk about this even contextually from a comparative point of view within different industries. So it's not just saying, hey, does one organization as a services company spend more or less than another, but also how are we compared and spending, you know, against, let's say, like a tech company who's, you know, got a bunch of engineers and doing the coding there. And I think what we're looking for is a what's like the baseline, right? So it's, it's benchmarkin…

AI assessment note: “I don't think there's a winner and loser That's going to be defined by did you token max”

Q little bit about agentic technology, agentic AI. Um, I'd love to hear your perspective just on, um, first of all, define what an agent is if you could, and then I'm curious to hear If these things actually exist, how far are your clients in, sort of, putting them into action? Or are we still just at this, like, time where it's just talked about but not a real thing?

A Yeah, so, um, as you mentioned, I do tech, media, and telco. I'm gonna use media as my example for what an agent is. In the movie business, actors have agents, right? And what do the agents do for them? The agent's job is to make They're actor employable, right? They're gonna go find scripts for them, they're gonna go get them employed, they're gonna go find other deals for them to do, it's advertisements for cologne or cars or whatever it might be, right? Um, you've given them the authority to act on your behalf in a specific area, right? Some level of agency, you know, for you. Take that concept of agency, apply it to technology. You're telling a piece of technology, in this case, almost like, you know, sass in a box, like this little mini, this little mini person who's sitting there, You're giving it authority to go do certain things for you. In this case, it's going to operate within a, you know, relatively closed environment. Perhaps it's on your desktop, or perhaps it's within your, you know, your platform application ecosystem, something like that, but you're giving it the authority to operate in there, and you're hopefully giving it rules to operate under, and you're also giving it skills. Skills might tell it like where to get information or how to do certain things, and then of course you're building security around it too. I love all the stories about agentics, wheth…

AI assessment note: “You're telling a piece of technology... You're giving it authority to go do certain things”

Q of like of skydiving in a way where you jump out of a plane and you're like, whatever happens, but I hope there's a system ready to catch me. From your position, you know, you're deploying agents in pretty high stakes moments. Even if you have the best governance in place, you have to be okay to a degree. Of losing control. So how do you become comfortable with that?

A That's, it's funny, right? I mean, I think, um, how much can you really control? Like in the, in the engineering space where you have the people like doing the code for you or in our space where you have the individuals doing the services, they're preparing it, you know, an audit or a tax return or a deal report or what have you. We're putting trust in these folks, many of you whom, you know, at like a superficial level. Um, When you start layering technology into it, because my view is that it's an augmentation of those people to make them better, not necessarily seeding all the control from them and putting into something that I know less of. That makes me feel a lot more comfortable. If you start getting into the space where a hundred percent of all the things that we do around, I don't know, let's say like booking travel becomes, you know, agentified, right? You just. Put in the query and the travel gets booked. I think that's where you do get uncomfortable. Like I appreciated that two nights ago, knowing I had to get back to hang out with you, Alex, and to make grandma's thing this morning, like I can ping my EA late at night and say, Hey, I need some help. Like I need to make sure with tornadoes coming through Chicago, there's going to be at least a plane on Thursday morning that takes off at six AM that could get me to the West Coast. If I'm pushing that into an agent an…

AI assessment note: “augmentation of those people to make them better... That makes me feel a lot more comfortable.”

Q token maxing. It's called token minimizing. And you're seeing companies like AT&T and Meta get really serious about cost. And we also know that Uber, of course, spent their entire budget in less than half a year. Who do you think is going to win out at the end of the day, the token minimizers or the token maxers? Given the definition of token max and you just gave us.

A Yeah, I mean, I think here's the good news. The good news is I don't I don't think there's a winner and loser that's going to be defined by did you token max or not token max? I think the winner and loser is going to be defined by did you outcome max or not? It's in part going to be a function of how did you incentivize people, which goes into that leaderboard and the things that got a lot of folks will say in trouble or certainly in the news, right? So there's that piece of it. It's the how do you actually want to encourage people to do it? Without encouraging the wrong things, it's take it too far, et cetera, right? So there's that piece or spend too much money. And then the other part of it is going to be actually, I think, from a planning point of view within an organization. And I deal with this as well. So I sit on the boards for for our US and our global organization at PwC, and we talk about this a lot. In fact, we talk about this even contextually from a comparative point of view within different industries. So it's not just saying, hey, does one organization as a services company spend more or less than another, but also how are we compared And spending, you know, against, let's say like a tech company who's, you know, got a bunch of engineers and doing the coding there. And I think what we're looking for is a what's like the baseline, right? So it's, it's benchmarkin…

AI assessment note: “I don't think there's a winner and loser that's going to be defined by did you token max”

Q What do you think about? Aaron Levy was here like 10 minutes ago, and he talked about how token maxing was a BS media narrative effectively, and it never really happened. Do you agree with that?

A I don't know if I totally agree with it. We were actually talking backstage just before he came on, and I think there's absolutely like we'll call it above above the, you know, above the plane sort of commentary that's out there that we're all seeing in both the mainstream media as well as like what Plays really well on social media, on X and other places and in podcasts. And then there's there's the reality within a lot of organizations. But it's it's happening enough where the behaviors, I'll call them, are slightly problematic from a cost and from an ROI point of view that it's real, right? You can't say that every circumstance is a problem necessarily, but it's coming through in a way that, you know, it's enough to think about and say, hey, are we doing this the right way, right? Broadly speaking, from an enterprise strategy and also then from Even a broader ecosystem point of view.

AI assessment note: “I don't know if I totally agree with it.”

Q What do you think about? Aaron Levy was here like 10 minutes ago, and he talked about how token maxing was a BS media narrative effectively, and it never really happened. Do you agree with that?

A I don't know if I totally agree with it. We were actually talking backstage just before he came on, and I think there's absolutely like we'll call it above above the, you know, above the plane sort of commentary that's out there that we're all seeing in both the mainstream media as well as like what Plays really well on social media, on X and other places and in podcasts. And then there's there's the reality within a lot of organizations. But it's it's happening enough where the behaviors, I'll call them, are slightly problematic from a cost and from an ROI point of view that it's real, right? You can't say that every circumstance is a problem necessarily, but it's coming through in a way that, you know, it's enough to think about and say, hey, are we doing this the right way, right? Broadly speaking, from an enterprise strategy and also then from Even a broader ecosystem point of view.

AI assessment note: “I don't know if I totally agree with it.”

Q Right. But that sort of goes to the way that the foundational labs and the cloud players are working with you. The things that I've heard is that these companies are making it so that when you plug into their systems, you're going to spend a lot of tokens and they don't really want you to be able to measure them. Are you finding that?

A So the I think the big counter to, you know, the the maybe the marketing and sales approach there is in the control plane that's being used. To actually help companies make decisions on what models being used or what interface is being used to do specific activities. And this is where this concept of central planning, and I say that as someone who's traveling to China next week, this concept of centralized planning within an enterprise is going to be so important. It's the you are not allowed to check the weather five times, five times a day using, you know, Claude 5.5, you know, 5.7 or whatever it might be, right? Like it's unacceptable, right? I would admit those is not a good use case. For weather checking, um, even if you're worried like I was last night about the tornadoes going through the Midwest and trying to get back here. So I wasn't late for Alex. It's still not a good use, right? You can do that still with the weather app, or if you really want to use something inexpensive, like there's a lot of those options out there. I think what's going to happen, you know, the term control plan sort of out there. It's relatively new ish. It's got governance elements. It's got cost control elements. It's got access to data elements. It's got, um, I'll see even like the human interaction elements, like what can you put into it? Not only what you can get out of it. That is going t…

AI assessment note: “the big counter to... the marketing and sales approach there is in the control plane”

Q But that sort of goes to the way that the foundational labs and the cloud players are working with you. The things that I've heard is that these companies are making it so that when you plug into their systems, you're going to spend a lot of tokens and they don't really want you to be able to measure them. Are you finding that?

A So the I think the big counter to, you know, the the maybe the marketing and sales approach there is in the control plane that's being used. To actually help companies make decisions on what models being used or what interface is being used to do specific activities. And this is where this concept of central planning, and I say that as someone who's traveling to China next week, this concept of centralized planning within an enterprise is going to be so important. It's the you are not allowed to check the weather five times, five times a day using, you know, Claude 5.5, you know, 5.7 or whatever it might be, right? Like it's unacceptable, right? I would admit those is not a good use case. For weather checking, um, even if you're worried like I was last night about the tornadoes going through the Midwest and trying to get back here. So I wasn't late for Alex. It's still not a good use, right? You can do that still with the weather app, or if you really want to use something inexpensive, like there's a lot of those options out there. I think what's going to happen, you know, the term control plan sort of out there. It's relatively new ish. Um, it's got governance elements. It's got cost control elements. It's got access to data elements. It's got, um, I'll see even like the human interaction elements, like what can you put into it? Not only what you can get out of it. That is goi…

AI assessment note: “the big counter to... the marketing and sales approach there is in the control plane”

Q Yeah, you're probably far ahead than most folks. How do you determine the ROI and and what percentage of your projects would you say are actually generating a positive ROI?

A Yeah, well, I'll tell you this. I know, I mean, even from an external point of view, I think MIT did did a recent publication talking about like what sort of activities are really, you know, replaceable just with generative AI in general, especially like in the vision and, you know, human interactive space. And I think they came up with an outcome of about 23%. As in, you wouldn't use a human to do 23% of the work. I look at that and say, in our business, it's probably not as high as 23%, but it's going to be some percentage of every single thing that people do. Right. So it's it's a different mathematical equation. It is absolutely going to be measured for our business. It's going to be measured in hours. The same thing that's going to be done in the engineering space for a lot of these companies as well. It's a measurement of hours. It's are you making the person more efficient? Are they creating more output? You know, for the amount of time they're spending on it, right? So is that lines of code they're producing or is it quality product, which this is an interesting thing, right? Like I could build a larger slide deck or I can write way more lines of code, let's say. But am I actually getting a product at the end of the day that people are saying, oh yeah, that's something I'm willing to use. And I had that conversation a bunch actually at a tech week in New York a couple o…

AI assessment note: “It is absolutely going to be measured for our business. It's going to be measured in hours.”

Q Yeah, you're probably far ahead than most folks. How do you determine the ROI and and what percentage of your projects would you say are actually generating a positive ROI?

A Yeah, well, I'll tell you this. I know, I mean, even from an external point of view, I think MIT did did a recent publication talking about like what sort of activities are really, you know, replaceable just with generative AI in general, especially like in the vision and, you know, human interactive space. And I think they came up with an outcome of about 23%. As in, you wouldn't use a human to do 23% of the work. I look at that and say, in our business, it's probably not as high as 23%, but it's going to be some percentage of every single thing that people do. Right. So it's it's a different mathematical equation. It is absolutely going to be measured for our business. It's going to be measured in hours. The same thing that's going to be done in the engineering space for a lot of these companies as well. It's a measurement of hours. It's are you making the person more efficient? Are they creating more output? You know, for the amount of time they're spending on it, right? So is that lines of code they're producing or is it quality product, which this is an interesting thing, right? Like I could build a larger slide deck or I can write way more lines of code, let's say. But am I actually getting a product at the end of the day that people are saying, oh yeah, that's something I'm willing to use. And I had that conversation a bunch actually at a tech week in New York a couple o…

AI assessment note: “It is absolutely going to be measured for our business. It's going to be measured in hours.”

Q Yeah, you're probably far ahead than most folks. How do you determine the ROI and and what percentage of your projects would you say are actually generating a positive ROI?

A Yeah, well, I'll tell you this. I know, I mean, even from an external point of view, I think MIT did did a recent publication talking about like what sort of activities are really, you know, replaceable just with generative AI in general, especially like in the vision and, you know, human interactive space. And I think they came up with an outcome of about 23%. As in, you wouldn't use a human to do 23% of the work. I look at that and say, in our business, it's probably not as high as 23%, but it's going to be some percentage of every single thing that people do. Right. So it's it's a different mathematical equation. It is absolutely going to be measured for our business. It's going to be measured in hours. The same thing that's going to be done in the engineering space for a lot of these companies as well. It's a measurement of hours. It's are you making the person more efficient? Are they creating more output? You know, for the amount of time they're spending on it, right? So is that lines of code they're producing or is it quality product, which this is an interesting thing, right? Like I could build a larger slide deck or I can write way more lines of code, let's say. But am I actually getting a product at the end of the day that people are saying, oh yeah, that's something I'm willing to use. And I had that conversation a bunch actually at a tech week in New York a couple o…

AI assessment note: “It is absolutely going to be measured for our business. It's going to be measured in hours.”