The Exchanges

Q you know, there's, there's a clear emotional aspect to this, which may or may not be present in other parts of the, of the business world. Uh, I think you mentioned the term empathy. How do you think about that? Like, sort of, you know, almost, like, emotional customization. Is that building the foundation models? Is there something that you build on top to, to do a good job there?

A Yeah, we, we, we do out of the box, um, make ADA as a sort of default experience, uh, as empathic as we, as we can. And the reason for that is, again, if we're focused on resolving a customer's issue, key to resolving a customer's issue is ensuring that, you know, you under, you convey that you understand their issue. Um, it's not sufficient just to do it unless you explain, you know, that, that you, you, w what you did and why. And so empathy, I think, is inherently connected to that. Um, I will say that we do see our customers customize that a fair amount. So the way that a business empathizes starts to really touch on the, you know, the, the brand personality of a company. And different companies, you know, like to handle that, handle that differently. Some like to be quite verbose in how they empathize, you know, others like to, um, you know, sort of empathize after the fact. They just want to get to it, you know, resolve the issue and then explain what they did afterwards. Um, I think it depends a lot by the, uh, on the business. I also think that over time we'll, we'll see like this dimension of, of ADA Like, really become personalized on an individual user basis.

AI assessment note: “we do see our customers customize that a fair amount”

Q So you've been doing this for, for a while now across a bunch of different customers. What, what have you seen, uh, Works and doesn't work in terms of accelerating that progress that you just described. In particular, uh, you know, the concept of onboarding AI, like how do you accelerate that?

A We observe effectively the companies that are maximizing, achieving the best results. They're going through a cycle of a couple different phases. They are, um, as you, as you mentioned, they're onboarding a new agent. They are, uh, measuring its performance routinely. Um, they are Coaching it to improve and then they're extending it into new places, whether that's, you know, in front of more of their customers within the channel that are first deployed or across all other channels or, you know, building it natively into their application if they're a software, software business. So it's really that, that loop of, um, you know, measure, coach, improve after they've onboarded that we see, um, you know, really the speed at which companies move through that is, is, is a, A key predictor of their overall results. Um, I think one thing I've learned over the last couple years of helping businesses operationalize this has been on the onboarding dimension. We are now very focused on making sure that the first day of your new customer service employee, like on day one, that employee is amazing. Uh, it used to be the case. We provided like far more configuration and control. It was almost like, you know, you are going to, we're going to give you a platform. You are going to create the agent of your dreams on day one, and then you will launch it. Now we've discovered that we can just accel…

AI assessment note: “accelerate results far faster by Giving you an agent in a box that is great”

Q close, um, you know, it sounds like we're in an early inning of, like, a bright, uh, future, uh, that, you know, some may consider it scary, some may, uh, fully embrace, uh, so what, what, where do you think we are in, like, two to three years from now? Like, any kind of, uh, I don't know, thought prediction, whether for ADA, for the industry, things you're excited about?

A I, I'm really excited for, um, people, for the average person to have a very positive experience with a customer facing AI. I think, I think that will, um, that, that is going to come, uh, in the next couple of years, uh, potentially much, much sooner. But I, I think the, the reason I'm excited about that is the quality of customer experience that We, as a population expect from businesses, like we are going to raise the bar for ourselves, and AI is going to do that. Um, and the, the, the consequence of that will be that if you are a business that offers a low quality customer experience, you are in trouble. And so I, I, I predict that customer service will become a far more strategic investment for the average business uh, as AI Demonstrates to the world that the quality of customer experience that lies ahead is way higher than we've come to expect historically, and that's going to mean a really great thing for those leaders who take the bold step of deploying this stuff now.

AI assessment note: “I predict that customer service will become a far more strategic investment”

Q that seemed to be corroborated by Klarna. At some point, there was this very well-publicized moment when they were able to decrease very substantially the number of people that they used in customer service using a foundation model. But sort of fast-forward to today, it seems that that discussion is largely gone, and in fact, again, like, the space seems more vibrant than ever. What do you make of that?

A Yeah, I think definitely 12 months ago, a lot of discussions about, you know, do you have proprietary models? Is your business a thin wrapper? You know, how far down the stack do you go? And I think in the last 12 months, most of the market has realized that the value is beginning to accrue at the application layer. And that's because it turns out that it's very expensive and technically quite complex to control your AI, to measure its performance, uh, and to steadily improve it over time. And I think it's those three factors that are starting to really, I think, help folks realize that what matters most is the value that we're providing to the, to businesses, not so much how we provide it. Um, it's turning out that there's a lot of product to be built, a lot of software to be built to deliver that level of control, observability, and improvement. And I think Ada is a good example of that, but there are other examples in the market across industries, not just customer service, where we're seeing that play out too. Software development is the other, is the other one.

AI assessment note: “most of the market has realized that the value is beginning to accrue at the application layer”

Q earlier, uh, sort of feels like, uh, this whole world of customer service from a job perspective, a profession perspective is completely changing. Uh, so what is, you know, in today, but also, you know, two, three, five years from now, if I'm a customer service expert at any of those companies Monday or Square or, uh, ZoomInfo that, that we mentioned earlier, uh, what will my day-to-day job be?

A So I think that broadly the big shift that's happening is, is customer experience, customer service professionals are, uh, are evolving from, You know, reacting to incoming customer service inquiries to proactively managing them by, by really serving as the manager of an AI agent or a group of AI agents. The, the role is a quite, quite different. It, it's, it's the same in that they're focused on improving the customer service experience, but the way that they do that is much higher leverage. Um, and by virtue of, of their agent being so performant, Uh, it starts to become a far more creative role. Uh, they start to be able to be far more focused on, you know, not just driving the efficiency of their customer service operations, but, uh, generating new revenue and, you know, fundamentally increasing the, the customer lifetime value of their, of their customer base. Um, we spent a lot of time, uh, helping our customers build their ACX organizations and identify the right kind of talent and chart the career paths of folks.

AI assessment note: “proactively managing them by, by really serving as the manager of an AI agent”

Q Any word on, uh, yeah, generative replies, generative actions, what, what do those mean?

A So generative replies refers to our, our, our, our generative capabilities that allow your bot to be generating answers in runtime automatically in a manner that's sort of grounded in your knowledge that you've integrated. And so, um, this is, this is, it's resulting in customer essentially at 25 to 29, 30% Improvement, percentage point improvement in resolution rate. Um, so it's a really major set of capabilities, uh, that we're really excited about that have had a big impact on the customer experience. Generative actions refers essentially the same on the API side. So instead of managing your, um, your, your API driven workflows prescriptively, um, you're, you're essentially just configuring your one API integration once. And your AI is deciding, essentially, is generating a workflow in runtime and executing it in runtime, and that's resulting in a, um, a much higher percentage of conversations where you're taking action, and that's elevating the quality of experience as a result. So, um, that's been also, like, probably even more exciting than the generative replies side of things, but, um, I think just good examples of how Uh, our customers are increasingly using Ada to coach their AI to improve and not to manage the prescriptive nature of everything their AI does. There's a growing level of trust that they create, uh, between, uh, themselves and their AI as it becomes more…

AI assessment note: “generative replies refers to our... Generative actions refers essentially the same on the API side”

Q conversation ended, or it can mean that the customer hung up in, you know, in frustration, or it can mean that the problem was satisfactorily resolved, and, You know, you could make the argument that when you're on the phone with someone, the customer agent can sense how the conversation ended, but maybe AI does or doesn't, I don't know, but how do you think about the concept of resolution?

A We, we think about it a lot. Like at the end of the day, we, we, we think of Ada as a resolution company. So our, our North star as a company is to resolve a hundred percent of conversations. Uh, and resolution for us means that the customer got the help they were looking for. Like we actually satisfied the initial intent intent of the customer while conforming to the policies and guardrails of the business. Um, you know, this is a resolution in many ways, I think is an example of how AI is transforming business to consumer, the economics of, of businesses and consumers interfacing. Um, prior to language models, it was prohibitively expensive to understand the quality of your customer service. We probably You know, everyone's heard that, that saying, this call may or may not be recorded for quality purposes.

AI assessment note: “resolution for us means that the customer got the help they were looking for”

Q uh, not just for ADA, but like for the industry in general, there is, uh, still a concept of handoff to a human, uh, at some point. Um, so a couple of questions. One, uh, how does that work, and how do you know, other than, you know, when the, the customer asks you when to do that, uh, and two, is that something that you think disappears over time?

A So, yes. So I, I, um, this is interesting and very topical for us. So, Yeah, Ada is a, is a, is an AI native application that has no human customer service agents in it. So whenever a human customer service agent is needed, we will route, we'll hand off, to a human customer service agent who lives in an incumbent system, like a service cloud. Uh, and we will pass over all context. We'll make sure that customer service agent can seamlessly pick up the conversation where it was left off. As we've started to achieve higher and higher resolution rates, we're starting to challenge our idea that a handoff is a failure. Because for most of our company's history, it's been a failure. Like we, you know, if, if we have to involve a human, it's, it's on us. That's a mistake. Increasingly, particularly as we look at this, you know, remaining 15% of conversations, those conversations tend to be incredibly complex. And many of them actually, uh, require some level of human approval just based on the processes that exist inside our customers. And so increasingly we're starting to think less in terms of handoff and more in terms of human delegation. And we're very excited about that because what it means is that instead of just simply handing off a conversation to a human agent and, and having them figure it out, increasingly what we're going to be doing is we're going to be take, we're going …

AI assessment note: “increasingly we're starting to think less in terms of handoff and more in terms of human delegation”

Q And what about, uh, RAG and pulling information from customers, uh, databases? How does that work?

A That's one of the modules in our reasoning engine. So one of the steps that we will follow, uh, again, depending on the inquiry, is, uh, we will perform retrieval augmented generation to surface the relevant knowledge from your knowledge base or series of company policies that you've integrated into ADA. Um, You know, we, we are very focused on the accuracy of our generations. So, um, you know, we use language models as judges to ensure that your generations are grounded in the knowledge and policies that you've, uh, that you've connected to ADA. Uh, so RAG is relevant, uh, to that, uh, as well. Um, But, um, but yeah, we, we think of, we think of knowledge searching essentially as a, one of the functions that your agent might perform depending on what, what task it has at hand.

AI assessment note: “That's one of the modules in our reasoning engine.”

Q Is it part of your onboarding when you have a new customer, like you do, do you ask them, uh, okay, well, what, what, how do you want us to handle your customers? You long winded, uh, to the point.

A We actually offer like a bunch of different sort of out of the box personalities that businesses can pick from. Um, and then because of, uh, we call it coaching. It's one of the most powerful capabilities that it has right now is the ability for, uh, a, The ACX team to just in their natural language, uh, coach their agent to improve so they can offer specific instructions at a conversation level to behave this way differently next time, and then observe their agent perform that differently for the next customer, both in a real and simulated capacity. Um, and we see that, that empathy is something that is often coached, you know, On a, on a, on an individual basis. So, you know, in this situation, you know, make sure to be more empathic. Because I observed here, this is actually a, a mo, this is actually a big deal to us. You know, when, when a package is late, you know, we actually, as a brand, really want to make sure the customer knows how deeply, uh, apologetic we are. And we really want to make sure that we're playing back to the customer. Like, I know that this is upsetting to you. I can only imagine, you know, how upset I would be if I were in a situation, and here's what we're doing about it. That's an example of like, you know, empathy, customers dialing it up and dialing it down on individual basis differently.

AI assessment note: “We actually offer like a bunch of different sort of out of the box personalities”

Q So they went from managing humans to managing AIs?

A Yeah, or in some cases not managing humans at all. Just, not just, but, but being a, being a highly productive, highly empathic customer service agent to now managing an AI. You know, going from helping 20 customers a day to helping two million customers a day. And then I think one of the things that's, like, most interesting about this transition, and it kind of goes to that point of, You know, when you deploy customer-facing AI effectively, you end up talking to your customers more. The ACX team has access now to all this conversation data that didn't exist before, and so they, they are now increasingly informing company strategy. Folks are coming to them saying, you know, what do our customers want us to build next? You know, what do our customers think of, you know, this initiative? Like, You know, are we supporting our customers in Asia effectively? What do you think we should do? What are our customers? Like, there's long been this promise of, like, voice of customer. It's always been this, like, little thing. Like, voice of customers.

AI assessment note: “Yeah, or in some cases not managing humans at all.”

Q And to the extent you can talk about it, can you, can you mention specific models, some GPTs, some clouds, some, what else?

A Yes. Yes. So we work today right now, our, our ensemble is mainly, uh, a combination of, OpenAI and anthropic models, although there are some open source models that we're also leveraging. Um, we, uh, you know, really test those models according to the module or, or specific, uh, use case that there we're applying them to. You know, we, we always have at least a capability that is leveraging the deepest reasoning. Uh, and that's, that is connected usually to our planning module. So to the, The, the intelligence that's determining, you know, what creative plan to propose to resolve your inquiry. That's typically a, a reasoning based model. Um, I think we're very quickly, uh, about to reach a tipping point whereby the most performant reasoning model available in the world may be overkill for the average customer service use case. Sort of like, you know, you don't need a PhD To chart a path to resolve your, you know, order refund request, even if it is adhering to a bunch of unique company policy, although we'll see. Um, I think that's TBD. Uh, so that, that's, that's sort of, uh, you know, an example of, of what we might test against. One thing that is definitely true that we measure, uh, quite extensively is a model's ability to adhere to instructions. So we're, we are quite maniacal about, uh, instruction following. And testing a model's ability to execute on what you asked it …

AI assessment note: “our ensemble is mainly, uh, a combination of, OpenAI and anthropic models”

Q of alluded to it a little bit in the context of, um, employees at, at ADA, but there is this, um, concept of onboarding the model like you would a new employee. Um, so if I'm a customer and I'm, I'm, I'm trying to work with one of these AI solutions, whether homegrown or through a, uh, vendor like, like ADA, what, what, what does that, what does that mean?

A So I think, so the, the, the, the paradigm here is that, um, well, let me, let me zoom out here for a second. I think the, the, what does it mean to be AI native? Um, well, I, I alluded to this earlier when we were, we were chatting that like being an AI native company means that you are, you know, your humans are augmenting your AI model, not the other way around. But the reason, the reason for that is because I think that We're at this very profound moment in tech history where AI's impact is going to be as profound, maybe even more profound than the internet's impact itself. And the reason for that is because if, if the web made the cost of distribution essentially free for businesses, what AI is doing now is AI is making the cost of cognition essentially free. And so because of that, AI native applications really need to be treated as employees. This is a truly capable AI application really is no different than a labor source. That was hugely, it was usually formally connected to human cognition, but, but, but it just so happens to be, to be AI now. And so the best applications I believe are going to be onboarded in a manner that are not that dissimilar from the way you formally onboarded a human employee. Uh, in our case, we, you know, make it easy for our customers to onboard Their customer service AI. They teach that AI to read much in the same way that you'd give a new …

AI assessment note: “teach that AI to read much in the same way that you'd give a new employee”

Q No, and, and, no, no, it's super interesting. And then at a practical level, uh, on the ADA platform, you started, what, having, uh, multiple models behind the scenes at sort of the core, is that like some open AI, some homegrown, what, what's, uh, what's happening there at that level?

A Sure. So ADA, You know, initially started, um, you know, seven years ago, um, really as an NLU company. We had a, you know, a core model that you were, you were, um, you were always improving as a customer of ADA, um, was a, and continues to be an intent classification model. And we employed large language models, um, Uh, throughout our history initially to augment the folks who live inside of the AI coaches or bot managers who live inside ADA to manage and improve their bots over time. And our, the language models, the, the generative AI were being used to really enable them to be more productive, but not to generate answers in runtime, uh, for end customers like we do today. And so, um, today that has changed. Now, um, because of the large language, large language models have crossed a fundamental quality chasm, as we, as we see. Um, the best experience, the most highly resolving experience that you can build with Ada that's really, like, exceptional. It actually, um, has a large language model at its core that is, that is making, that is generating an answer and executing a workflow, um, in runtime so that every generation, essentially, every conversation between your brand and one of your customers is unique. And underneath the hood, there's, as you alluded to, there's a bunch of different models that are, that are, um, involved in that, and we take away the complexity for …

AI assessment note: “underneath the hood, there's, as you alluded to, there's a bunch of different models”