The Exchanges

Q What was sort of the dynamic to ultimately land on the date in the early spring, I think it was, that it ultimately shipped?

A The amazing thing about working on a product like Bard is like, that genuinely feels like three lifetimes ago. When I reflect back on the time, it was at what point will we create something in the U.S. and U.K. that will, for the most part, generate great experiences. We knew that code, for example, was something that people were using language models for, but we didn't think it was good enough, and so at some point we made the decision to say, look, code's not ready, so code launched a couple weeks after we launched, but we need to get this out Into the world and start experiencing what quote unquote real world, um, prompt streams will look like. And so I don't know that it was a specific date so much as it was, we need to start learning quickly and we need to hit a certain quality threshold that we are going to deliver while minimizing the, the safety risks. And so spending a lot of time going through red teaming, setting the policies, et cetera.

AI assessment note: “I don't know that it was a specific date so much as it was”

Q Do you think the type of product people that will be successful building in sort of with these new primitives Are the same type of product people that have been successful building all sorts of other stuff over the past decade, or there are kind of new wind conditions now?

A The thing that won't change about a great product leader is the one that is able to be precise in articulating their hypotheses of what they're trying to learn and insights and lessons translating to the feature set that you see. Like, I think every great product leader needs to be a deep product marketer at their core. I don't think that changes. What I do think changes is the, how do you do that while balancing the more probabilistic nature of the product? We've seen this technology really, like, it, it's a consumer discussion, but so many of the startups right now are in the enterprise space where there's a very different tolerance for probability and imprecision. And so finding the way of harnessing that as a feature rather than a bug, I think is going to be challenging, especially when one of the perceived shortcomings of your product is actually going to be, is like the core of what makes it function. I don't know that there's been technology in the past where that's the case. I'm specifically referring to the hallucination bit. Like, if you ask someone, how will humans get to Mars? If you're going at it from a path of like, Enterprises aren't buying this because it hallucinates. I'm going to build the most non-hallucinated answer. What you're going to get is, um, how will humans get to Mars? We don't know. Versus embracing the hallucination. How can I ground on what theo…

AI assessment note: “The thing that won't change about a great product leader... What I do think changes”

Q And so have you found that you could take somebody Who didn't move particularly quickly working on another product at Google, and in the context of this product and, and this team, are sort of night and day in that dimension?

A I've seen people who have been part of teams at Google that it's taken their teams two years to ship something, and when you hear two years, you might have that sort of, oh my goodness, there's just a lot of Existing user work that you have to go through. You have to ensure that if it's, uh, you know, a change in terms of service, that that's deeply understood. Like, these things are anchored in the respect for the opportunity that we have and the respect for the user that we have. Come to a project like Bard with a Let me help you anticipate what could slow you down in the future. So they're not coming at it from a perspective of, OMG, I can't believe I moved on a thing that took two years to ship. It's how do we create decisions now that when we fast forward 18 months from now, we won't be stuck in a position where it takes two years because we hadn't thought through some of those problems early on. And like, the people that have experienced Pain, in various regards, are the ones that have been by far the most successful.

AI assessment note: “the people that have experienced Pain, in various regards, are the ones that have been”

Q What was the process to figure out what the first version of BARD would look and feel like? It's tricky because you could be working on the first version of that product for two or three years, let alone the fact that you did it so quickly. So what did that process look like?

A The first version of any product , you have to enjoy using yourself. I don't care whether you're a startup that's building a record label in your pocket. Uh, one of the first things that was ever uploaded to United Masters when I was working on it was artwork that I helped create with a colleague of mine whose song, uh, it was to upload it, and we just went through the path of releasing music ourselves. Working with Bard is a very similar thing Topic, which is the first prompt that got put in was by our team trying to see how it feels, how it works. And the hardest part wasn't figuring out how it would respond in the sort of happy path. It's the, wow, these things can really get it wrong. So there's things that are obvious that, uh, or seemingly obvious In early versions, you don't want a nuanced answer toward. How do I commit a crime? Look, to get a very proper nuanced answer for that is going to be really hard. There's research that takes place. You have to take a step toward what we call adversarial safety risks to say no. The harder part are the inadvertent safety risks. Things that aren't intentionally going to be adversarial that can make your product go in a vector that you don't want to go. How much Tylenol should I give to my kid when they have a fever? Very well-intentioned question, but you want to make sure that the answer that you provide is as tuned toward not hal…

AI assessment note: “first prompt that got put in was by our team trying to see how it feels”

Q But in the case of that idea of let's explore what it would be like if the product did things for the user versus just collaborated with the user, what is the process by which you end up focusing on that versus 30 other things? Or if there's a series of hypotheses, how would you force rank or decide which hypothesis you want to test and validate first?

A This is where having research as part of your core product development process is key. You've got to generate insights in terms of something that your most passionate users are advocating for. People that have heard of your product but haven't, but haven't used because it doesn't do X. It's not a very precise equation when you do one or the other, but you're doing opportunity analysis in each of these cases and knowledge of what the technology can do. And so, what we're trying to do to determine what you build next is that what do we believe is going to have the most amount of impact in the clearest amount of time. So there are open-ended things and elements of research that, of course, we're looking into as a company. Things that are unsolved research problems will be unsolved research problems. Things that are more clear, even if it takes six months to do, but you think, oh, this will create a cohort of X number of people that haven't used the technology. That's what you use to kind of balance.

AI assessment note: “determine what you build next is that what do we believe is going to have the most amount of impact”

Q What did this whole experience sort of teach you about risk taking in the context of a very large sort of very valuable company? And I think it's particularly interesting because you worked at very small companies, um, and you've obviously worked at Google now multiple times.

A There's this amazing thing about Google specifically. After I left my first stint, I was in search of, and I was so happy that it was true when I came back. So my first day at Google was in September of 2007. And there's one of the core principles of the company is focus on the user and all else will follow. And what that manifests is a culture of curiosity. This sense of, what can we learn about the world that helps us achieve our mission? I've worked at many companies across Silicon Valley, and I've never been at one where I can recite the mission statement, organize the world's information to make it universally and easily accessible and helpful. That curiosity guides us of, we're out to achieve something bigger than The current products that people have to use today, whether it's ones that we've built or somebody else has built. But we genuinely believe that humanity will be in a better place when you have access to this information at your fingertips. And that has been true on Bard, and it's true across Google, that there is this curiosity of, wait, this fundamental truth of computing that's been true my entire life is no longer true. This truth that computers exist to do things for you. Give me the distance between point A and point B. Alright, here's the answer. Calculate this column of numbers. Alright, here's the answer. Now that's not computing's sole existence to fol…

AI assessment note: “it's a bigger risk to not understand this paradigm shift”