The Exchanges

Q looked was in the checkout process. How can we, you know, deploy this there? And, you know, for me, it seems interesting because intuitive, like, like going by my gut, I would say maybe at the beginning when you're in discovery or at the end when you have customer service issues, but you're saying at the checkout. So what kind of questions come out at checkout and why start there?

A Um, we felt that the checkout was really the fulcrum point where customers have the most questions about a booking. Like they've already are knee deep into the details page of one of our products. So say it's a hotel and they're about to purchase something, but they have some, you know, just nagging questions that could be plaguing them. Like a great one is, is it pet friendly? Um, what are the points of interest close to this hotel? Uh, what's the neighborhood description around it? You know, answering those final closing details for the, for the consumer builds more confidence in their booking, and that's why we, we started with, with checkout from, for, from our perspective. Two, it's at the heart of what we do. Uh, so creating a frictionless checkout experience that answers all questions for our customers before their booking, we felt was a great place to start.

AI assessment note: “we felt that the checkout was really the fulcrum point where customers have the most questions”

Q may, it would be amazing to say, like, given your past, uh, uh, bookings, we know you like places with good Wi-Fi, generally downtown, close to a good coffee shop, you know, ratings above Nice to see you again. Here's what we found. Um, as opposed to me having to reintroduce myself over again. So where are we in the state of like being able to get to that point?

A Well, I think this is more about how companies design and develop their data architectures than generative AI understanding personalization on the individual level. So over the last several years, we've made huge investments in building out a customer data platform that actually understands who you are, what your booking preferences are, what you've searched, what you've booked. When you've booked, have you booked as an individual traveler or have you booked as Traveling with a family and what have been the nuances of hotels and flights you've taken as an individual versus as a family. And so what we've done is we've married all that rich CDP data so that when you log into the gender of AI experience, a signal is fired that understands who you are. So in this case, you're Alex and I understand what you searched on, what you've booked in the past. What are your preferences? So that as we're going through the conversation, it completely understands the background that you have as a customer based on all the intelligence we built into our data infrastructure. So that way it's a very personalized experience, um, in that conversation to understand all of those preferences.

AI assessment note: “we've married all that rich CDP data so that when you log into”

Q in your team? And they say, well, we're, we're still getting to that. So You know, I have a chief technology officer of a major company sitting before me who's working to put this into practice, and I'm going to put you on the spot with that question and say, how's that going for you? Um, and, and have you been able to see those results in a concrete way?

A So we've just recently completed a pilot of a product, uh, that played in the generative AI space as it pertains to code automation. And quite frankly, we, we had mixed results. We had certain pieces of functionality that were very productive, productive for our, uh, our engineers, such as quality, you know, automation of, you know, you know, unit tests or suggestions in code. So those were very positive. On the mixed bag results side, uh, one of the striking things that we saw was Was a developer ready to use the code generated by the AI model as a commit into production? And There was still some hesitancy around doing that. Um, and I think it's really two factors. Number one, really trusting the technology to complete a piece of code that you're responsible for and commit it to production. And two, it's getting developers used to working in a pair programming model where the other person you're working with is a large language model, which some developers have not Wrap their head around yet. So, um, you know, we've seen mixed results. Um, and I think the jury's still out on the proliferation of using this capability to fully automate what a developer does.

AI assessment note: “we've just recently completed a pilot... quite frankly, we, we had mixed results.”

Q line. Uh, I can't even imagine the scope of content that you're working to create. And so that's why I want to ask about it. Um, How does this help? What type of content is it? Is it creating? Um, how are you again, preventing from hallucinations from taking hold and how are people responding to some of the content that you're using generative AI tools to put out there?

A So the, the marketing content is all about creating merchandised pages, uh, blogs, um, detailed ad snippets, you know, profiling, uh, a descriptive, um, City, for instance. So we've, we've taken the top 1000 city destinations in the world and have been systematically creating content in those three different channels I, I talked about. You know, a part of, a large part of our, our business is, is indirect business that comes through people searching. So providing very high quality merchandise, personalized content around these cities We felt was a great way to apply generative AI. Uh, on the second, on the second part of your question around hallucinations, uh, we've built a hallucination service that verifies the information using third-party information sources. Like, um, in the case for a lot of our, our hotel products, we're using Places API from Google. Um, which is really an authoritative source for a whole host of geographical, uh, information. And so as part of our pipeline, there's a hallucination check that occurs to ensure the accuracy of the information that we're given. We also have another check in that process, which is really human copy review, where we have humans review the copy, uh, being generated after our hallucination check, just to make sure That there's a level of accuracy for the information that's going out.

AI assessment note: “we've built a hallucination service that verifies the information using third-party information sources”

Q you with information. And there's only so much information you can hold on like the front page of a, of a travel website like Priceline. So does, does, um, the, the search functionalities that you've, that you're enabling with this bot, does it Offer a surprising or like deep, deep down in the data results that people can, you know, end up making better choices with when they encounter it?

A Well, one of the, one of the great things around generative AI is that you can marry it with your own data sets internally, which, uh, for us has been game changing. So what, what we're doing within our platform is we're taking all of our hotel content And putting it through a machine learning pipeline and marrying that data, our own data, with authoritative sources of location information and map information to provide complete detailed accuracy on things such as hotel reviews, neighborhood descriptions, points of interest. The second advantage to taking this approach, it also reduces The amount of hallucinations that you would get from a large language model. So marrying the large language model with our own data sets with third-party data really drives down, ah, the amount of hallucinations that we get from these kinds of conversation and almost creates authoritative sources of cash information around hotel reviews, points of interest, Neighborhood descriptions, um, that allow the site to perform faster. Because the other thing that to, to take into account with all of these things is the cost. Um, cost in, in running and managing a generative AI platform could skyrocket. So you have to be very smart about when you refresh or make direct calls to the large language model and this third party information.

AI assessment note: “provide complete detailed accuracy on things such as hotel reviews, neighborhood descriptions, points of interest”

Q people are thinking I can do my entire itinerary just by saying to a chatbot, hey, you know, find me some island in Greece that I can stay at for seven days for, you know, 2000 dollars a person. I'm just throwing numbers out there. So I'm curious from your perspective, you're the one that's going to, you know, help enable this. Where does your thinking land on that front?

A Well, we're a very data-driven company, so we started embedding generative AI capabilities right into our checkout flow, where if a customer had a question around a hotel, a flight, a product, that before they booked, that they could have the experience of asking questions about the product that they were potentially booking, and so we, we dove right into the heart of our customer journey, went right to the checkout flow, where not only Would we enable, uh, customers to ask questions and get answers about the products that they were about to book, but also embedded the checkout flow right within the experience of generative AI to allow them to have a frictionless experience that once all their questions were answered, that they could immediately just go to purchase. Uh, so that was where we started. And more importantly, because we're highly AB, uh, AB testing driven, We ran tests to make sure that the bot had a positive impact on the user experience and the, you know, and revenue, and after several iterations of prompt engineering, we got to a point where the bot was A-B testing positive, and so we pushed it out to a hundred percent of our, our, our customers on our platform. The next part that we tackled, which we recently just launched, is Putting generative AI into the post post booking process to answer any questions that customers may have around any of the products that …

AI assessment note: “Now we shifted our focus towards what we're calling The travel concierge part”