The Exchanges

Q how do you know how much data is enough to make a decision post a launch? You know, is it two weeks of data where you're like, okay, we see the writing on the wall. Sometimes it just takes longer. Like we said before the show about consistency of content creation. How do you know post product launch when enough data tells you the answer versus when you need more?

A Yeah. I always like to start with before you launch, before you even start building, what is success? Like, like, you know, once you launch it, what should be going up into the rights that should be excited about? Um, don't tell me in retrospect because, you know, then you're just trying to fit the data into what you're trying to do, but tell me ahead of time. And it's okay to, again, you might be wrong, but not confused. Just have a strong opinion around it. So we can see if we're building towards the intuition we have. I think for me, it's, it's, there isn't a notion of enough data is that the hypothesis you have isn't being validated. Like ultimately you have success with a product when you have real adoption and retention. People actually find product market fit. They're coming back to the product to engage with it. That's when, you know, you really have something, uh, and adoption by itself, you know, if you have large enough base could also be, uh, in a way gamed in a while because discovery is really strong and you can always get people to try out something, but will they stick around? Will they come back to use it? That, that is your real test. We do a lot of processing around the notion of evidence versus conviction. You can build something with strong evidence that you had before, or you can build it with conviction, with a hypothesis of showing evidence at a certain …

AI assessment note: “there isn't a notion of enough data is that the hypothesis you have isn't being validated”

Q You said very early there in terms of where we are. I just have to ask you. Elon said we cannot let it out of the bottle because unlike most technologies before you can put, you can put most of them back in the bottle. You won't be able to revert this progression. Do you agree with him? And how do you think regulation looks for AI moving forwards?

A I think when you think about this technology, uh, your spectrum for me, you know, lies from high excitement to concern as well, because it is like with any powerful technology, it could be used to do amazing things and as an amazing productivity tool. And it could also be used as a weapon. Uh, so the, the notion of being extremely responsible with how this technology is being used, especially for the ones building it, I could not agree with more. It's, it's how, it's how I think we build and how we interact. We know we've been investing For a while now, even long before this technology in what we call responsible AI principles, which is how you build with transparency, how you build with inclusivity, how you build with privacy. You make sure it's built into the models. It's built into the interaction and you don't release something to the public unless those principles are being met.

AI assessment note: “we've been investing For a while now... in what we call responsible AI principles”

Q That's a really interesting one, which is that you mentioned revenue there. How do you, as a CPO, think about products which generate revenue today, which is very important. Um, but also versus the challenge of investing in innovative products for the future with unknown upside and untangible value creation. How do you balance those two?

A Yeah. So ideally, you know, everything we do starts from how it led us up to our vision. And this is something that I cannot emphasize enough. Like it's, it's a credible part of how we think of like, then we literally start our sessions and our planning process for what's the vision for the company. But then that translates usually to a strategic plan that spans between one to three years. And that plan usually have very specific goals for where do we want to drive revenue generating, you know, in the, from an annual perspective, like let's call it the short term annual perspective. And what's the long-term innovative products we're trying to build? And that plan usually encompasses of both. Um, so for us, like it's the combination of both the annual plan that used to basically has a revenue target on it, all the way to the long-term innovative products that we basically have there. Now, that said, we operate in a very competitive, fast-pacing environment. So it's not like we have a three-year plan, quarterly plan, and we go and execute. We do continuous planning. Like we do, as soon as strategies evolve, we basically run it.

AI assessment note: “That plan usually encompasses of both.”

Q That's a really interesting one, which is that you mentioned revenue there. How do you, as a CPO, think about products which generate revenue today, which is very important. Um, but also versus the challenge of investing in innovative products for the future with unknown upside and untangible value creation. How do you balance those two?

A Yeah. So ideally, you know, everything we do starts from how it led us up to our vision. And this is something that I cannot emphasize enough. Like it's, it's a credible part of how we think of like, then we literally start our sessions and our planning process for what's the vision for the company. But then that translates usually to a strategic plan that spans between one to three years. And that plan usually have very specific goals for where do we want to drive revenue generating, you know, in the, from an annual perspective, like let's call it the short term annual perspective. And what's the long-term innovative products we're trying to build? And that plan usually encompasses of both. Um, so for us, like it's the combination of both the annual plan that used to basically has a revenue target on it, all the way to the long-term innovative products that we basically have there. Now, that said, we operate in a very competitive, fast-pacing environment. So it's not like we have a three-year plan, quarterly plan, and we go and execute. We do continuous planning. Like we do, as soon as strategies evolve, we basically run it.

AI assessment note: “translates usually to a strategic plan... And that plan usually encompasses of both.”

Q Yeah, no, I, I agree with you. Toma, uh, tell me, if you could change one thing about the LinkedIn product today, what would you change that you haven't already?

A Yeah, I think, you know, well, what, what, what's, what's interesting right now, we talked about the idea of UI and complexity. There's this great law that I'm, uh, I really like. It was really helpful early in my career to think about how to build products, which is the conservation of complexity. The idea that every product has an inherent amount of complexity and the idea is you can, you solve it In the product development side, or do you actually put it on the user to solve by themselves? And I think we're in this phase right now where there's so many use cases and audiences that people come to the LinkedIn kind of main app for, and that just had inherent complexity because there was so many, you know, you come in the morning, you're trying to see what's happening in the world. Later in the afternoon, you interview somebody, you want to check out their profile. Then, you know, your boss upsets you later in the day and you want to see what's out there for you. This is all three use cases in the same Like, 24 hours. I think now with AI, and I would claim probably this would be a more generalized statement, the more complex your product, the more impact AI could have in terms of simplifying it for your user base. So one thing that we're already underway for us is to really take away the complexity and solve it with the idea of bringing in more of this new, uh, large language m…

AI assessment note: “take away the complexity and solve it with the idea of bringing in more”

Q years of healthcare data that you can, you know, bluntly utilize the models on. Where it then is actually 30% open AI or 40%, whatever it is, and 60% new. But this thin layer of, I'm seeing so many sales tools, onboarding tools, human, or I don't think value accrues onto that. And so I'm like, where does value accrue then in this wave that you mentioned? I'm an investor.

A So if you ask me a couple of years ago, I would have told you that data is everything in AI, right? Because computing power is accessible and, and the models are accessible to all the open source. So it's really about the data you have. And, and there was a, you know, people coined data as the new oil, and there was a reason for that data was the fuel. That helped AI become better, and then the more data you had, the better products you can build, the better products you have, the more users you have, the more data you have, and you have this virtuous cycle of success. It was also vicious cycles for startups. A startup has to innovate around how do we get data, and there's ways to do that as well, by the way. Some very amazing scrappy startups were able to bring data in a very innovative way that helped them compete. Now, what's happening right now is, you know, technologies like GPT are already trained on all public data. They've read every book, every healthcare book. They've trained on every piece of literature, art, best practices. This, the whole idea of the pre-trained is one of the biggest inflection points of this technology. It's already trained on every available public information out there. So the data advantage that used to exist a couple of years ago, I think is getting diminished. But not to a full extent, because to your point, you can still bring your own propr…

AI assessment note: “the data advantage that used to exist a couple of years ago, I think is getting diminished.”

Q how do you know how much data is enough to make a decision post a launch? You know, is it two weeks of data where you're like, okay, we see the writing on the wall. Sometimes it just takes longer. Like we said before the show about consistency of content creation. How do you know post product launch when enough data tells you the answer versus when you need more?

A Yeah. I always like to start with before you launch, before you even start building, what is success? Like, like, you know, once you launch it, what should be going up into the rights that should be excited about? Um, don't tell me in retrospect because, you know, then you're just trying to fit the data into what you're trying to do, but tell me ahead of time. And it's okay to, again, you might be wrong, but not confused. Just have a strong opinion around it. So we can see if we're building towards the intuition we have. I think for me, it's, it's, there isn't a notion of enough data is that the hypothesis you have isn't being validated. Like ultimately you have success with a product when you have real adoption and retention. People actually find product market fit. They're coming back to the product to engage with it. That's when, you know, you really have something, uh, and adoption by itself, you know, if you have large enough base could also be, uh, in a way gamed in a while because discovery is really strong and you can always get people to try out something, but will they stick around? Will they come back to use it? That, that is your real test. We do a lot of processing around the notion of evidence versus conviction. You can build something with strong evidence that you had before, or you can build it with conviction, with a hypothesis of showing evidence at a certain …

AI assessment note: “there isn't a notion of enough data is that the hypothesis you have isn't being validated”

Q the craftsman, uh, and then there's the operator. The kind of craftsman is kind of the in between of the visionary and the, um, and the operator. And the operator is like your scaled CPO where bluntly they instill processes They're kind of the professional CPO that puts processes and infrastructure in place. Which would you say that you are of the three? And would you agree with those characterizations?

A I think those attributes of what's expected in the role, uh, in the role are correct. Um, and I think it's hard to find somebody who is locked immediately at the center. This is one where I would send you to my team and people will work with me and say, I would, you know, curious to say what they say. You know, talking about myself, I, you know, if I go back to why I love building, I love all three attributes. So I can tell you what I love doing, but I let others tell you what I'm good at. Um, I love the imaginary side. I love the vision side. I love getting excited about what the change we can create in the world if we build something pretty unique. But then I love taking it all the way to how we actually execute on it. I love the grind of the work itself. So I don't stick around with just, uh, you know, a great whiteboard session. I love seeing it in action. I love feedback. I love seeing how, how you can take something that, you know, most launches are never zero to one. They're like zero to .6. They're never great. And like moving it from .6 gradually to really finding product market fit. That's an incredible feeling. When I had my startup, knowing that somebody's using something you built from scratch and something you invented is an incredible experience. It's, it's one that I can tell you, I can see if you love product or not, if you get excited by it. For me, it's less …

AI assessment note: “I can tell you what I love doing, but I let others tell you”

Q of the internet. Sounds great, but guess what? It means fuck all to that SMB who just needs to make 10% more money so they can buy that coffee machine. The GDP of the internet? Intellectual. Too much. Like, do you worry sometimes that LinkedIn at the scale and incumbent size you are, you almost lose product messaging touch with fuck. I actually just need another client as a freelancer.

A It's, it's a wonderful point and one that, uh, kind of touches on why in product jams we emphasize the job to be done you're trying to hire for, you're trying to do for. So for example, even if you're solving, you're saying, even if, for example, you selected the job to be done for a job seeker, there's so many jobs to be done for a job seeker. You could be building, you know, a job seeking experience for people who are looking for hourly work. You're trying to find a job for people who are, you know, playing more in degrees that, you know, they're more professional hires in the market itself. You're trying to find jobs for what we call first line hires or people out of school. We're trying to find for the, for the same time. So for me, being able to talk specifically, not just on the audience and the overall arching, um, Value proposition you're trying to deliver for them, but going very narrow to the problem you're trying to solve is key. When you have that, then you're not trying to solve for everybody. You have very specific jobs to be done, you're solving for people, very specific audiences you're solving for people. Now, all that is up to the vision. So after a while, once you think you've done a good job with that job to be done, you can move to the next level. So for example, at LinkedIn, one of our fastest growing segments are Gen Z's and entry level professionals. So …

AI assessment note: “It's, it's a wonderful point and one that, uh, kind of touches on why”

Q Does it change how you do anything with regards to data collection, data treatment, data cleansing?

A I love that. Ok, this is, I have to take this one. Ok, I'm gonna start. Um, this is, this is a big one for me. It's been a pet peeve of mine for a long time. I've been preaching, preaching like AI first mentality for a long time and really trying to push both design and product folks and even engineers to think about data collection and quality data for a long time. Again, this has been like a thought of being like, it's almost like, uh, somebody else will do it. Data science will figure out how to just collect it from the signals. And like, you don't understand this is, this is the literally, this is oxygen. This is what feeds If AI is at the center, what basically brings it to life is data. I remember like early on at LinkedIn, I used to literally like spend so much time filtering by myself data. So I can basically say, this is good food. This is good diet. This is what should eat for like, this is what the algorithm should, should eat. But somehow people gravitate towards the UI or they gravitate towards the specific individual elements of it. Not understanding that what actually fits the algorithm is the data. My opinion as a product leader or a designer or an engineer or a product manager, like data is literally at, um, your second most important job in your role. It's understanding how do you, how do you basically start to infuse all of data collection and make sure it's …

AI assessment note: “understanding how do you basically start to infuse all of data collection”

Q here. I'm going to get you in trouble with comms, Toma. But LinkedIn is a publishing engine as well. I publish to LinkedIn my posts. Many people publish amazing content to LinkedIn. OpenAI and ChatGPT in many of these models will be scraping and extracting your content and your value. Paying you nothing. How do you think this looks for content publishers moving forwards? What's the, what's the thoughts there?

A Yeah. I think LinkedIn is a platform, uh, that allows people to exchange information in many ways. We actually open up, um, discovery for information being shared on LinkedIn, because it's a great way for creators to be able to be discovered. Outside of LinkedIn and then, uh, they can build the reputation on LinkedIn and being know they can be discovered everywhere. I do think though, you know, there's the question around the role this technology is playing with publications in general. Like if now I can get an answer. For this tool, do I need to go all the way to the site to engage with it? And I think that, uh, that evolution was still being developed and built. I think we had the same thing with search to an extent where if I can get the answer for search, do I have to go all the way to the site and actually get the response in the site?

AI assessment note: “that evolution was still being developed and built”

Q Toma, we chatted before about content creation on the platform and how much I loved it. I think the shows are also successful because I'm also quite honest, more so now than I have been before. Can I ask a blunt question, which is when you look at the desktop site and you look at the mobile app, do you not feel that it feels outdated from a UI perspective?

A I think there's a lot that we can do there to improve, and there's that actually we're working on right now to really innovate within the constraints of what we've done. In fact, once you start playing with the app itself, there's a lot of innovation in between the feed, and we can go deep on the feed and the role it's playing today, and within the messaging experience, within the search experience. In fact, the use case of LinkedIn dramatically evolved over the last few years. But I do feel like in many ways, the current experience that's in constraints us, and there's a lot we can do to improve it and, and, and to rethink it. That's what's happening right now. In a world of generative AI right now, there's a lot of complexity you can potentially unwound. Uh, by building a simpler experience while playing AI into it. If you play the last six years for us at LinkedIn as a company, there's been tremendous evolution in the product that led to some pretty remarkable results. Revenue for LinkedIn more than tripled. Our memo base more than doubled. Our engagement is reading record levels. And I mentioned we're about to hit a 20 years old company in a couple of weeks. We're growing the fastest we ever grew right now. Which is pretty remarkable for a twenty-year-old company.

AI assessment note: “I think there's a lot that we can do there to improve”

Q You said very early there in terms of where we are. I just have to ask you. Elon said we cannot let it out of the bottle because unlike most technologies before you can put, you can put most of them back in the bottle. You won't be able to revert this progression. Do you agree with him? And how do you think regulation looks for AI moving forwards?

A I think when you think about this technology, uh, your spectrum for me, you know, lies from high excitement to concern as well, because it is like with any powerful technology, it could be used to do amazing things and as an amazing productivity tool. And it could also be used as a weapon. Uh, so the, the notion of being extremely responsible with how this technology is being used, especially for the ones building it, I could not agree with more. It's, it's how, it's how I think we build and how we interact. We know we've been investing For a while now, even long before this technology in what we call responsible AI principles, which is how you build with transparency, how you build with inclusivity, how you build with privacy. You make sure it's built into the models. It's built into the interaction and you don't release something to the public unless those principles are being met.

AI assessment note: “We know we've been investing For a while now... in what we call responsible AI principles”

Q Do we need to restructure the fundamentals of our teams for these new processes? So when we think about moving to these instance creations, and when we think about fine tuning our data on pre-trained models, but we're using our data, do we need scientists in our team? Do we need a completely different structure for how we build and develop products?

A I think you need new skill sets in your team, for sure. Um, you need people who are working very closely with this technology. I think AI talent is going to be one of the most important talents to have in every company. Uh, people who intimately understand how this works and can operate this technology. So one of the most important interfaces Is the prompts, right? What people use in ChatGPT as a fun way to write prompts. That's actually a really critical interface. That's the interface between the machine and the human. And being able to understand how you, this is how you talk to AI. And prompt by itself is a masterful skill today. If you master the work of, by the way, that's a, the best practices playbook is still being written because in many ways we're still almost like reverse engineering how this technology works. Uh, but the skill of understanding how to talk to AI, to your point, how to build my fine tuned data into the process, how to leverage it in, in, in the direction of building amazing processes is an extraordinary skill. I've been playing with, ah, with prompts for a long time, and, you know, the examples I give, which is just a fun example, I can ask AI, explain quantum physics for me. And it will explain it, and it will be an okay explanation, you know, very much a Wikipedia kind of like explanation. But then I can say, Hey, explain to me quantum physics as i…

AI assessment note: “I think you need new skill sets in your team, for sure.”

Q Does it change how you do anything with regards to data collection, data treatment, data cleansing?

A I love that. Ok, this is, I have to take this one. Ok, I'm gonna start. Um, this is, this is a big one for me. It's been a pet peeve of mine for a long time. I've been preaching, preaching like AI first mentality for a long time and really trying to push both design and product folks and even engineers to think about data collection and quality data for a long time. Again, this has been like a thought of being like, it's almost like, uh, somebody else will do it. Data science will figure out how to just collect it from the signals. And like, you don't understand this is, this is the literally, this is oxygen. This is what feeds If AI is at the center, what basically brings it to life is data. I remember like early on at LinkedIn, I used to literally like spend so much time filtering by myself data. So I can basically say, this is good food. This is good diet. This is what should eat for like, this is what the algorithm should, should eat. But somehow people gravitate towards the UI or they gravitate towards the specific individual elements of it. Not understanding that what actually fits the algorithm is the data. My opinion as a product leader or a designer or an engineer or a product manager, like data is literally at, um, your second most important job in your role. It's understanding how do you, how do you basically start to infuse all of data collection and make sure it's …

AI assessment note: “It's been a pet peeve of mine for a long time.”

Q Is net new knowledge not AGI? Is that not the ability to deal with ambiguity, the ability to make subjective decisions? Just, I'm, I'm purely asking. I'm not like posturing. I'm genuinely intrigued.

A Yeah. I'm hoping you're going to ask me to define AGI, uh, because that's a really hard task right now, but, but in a way it is a way you're trying to build as close to human intelligence as possible. And human intelligence, you know, going back to our conversation about art and science, like the ability to be imaginative, to hypothesize, to have a vision, that's what makes us, that's what makes us, you know, naturally human. That's our, that's our skill set. And it's not the day-to-day judgery that makes us human. It's the imaginative side of our role that makes us human and extremely powerful. So there is a point where you start asking if this, when this technology reaches that level, what is next? Then, and then it's not just going back to the drawing board. It's like, I don't know if there's a drawing board. Like you almost go back to rethinking what, what is the role of this technology and how do we really start interacting? The human and machine interaction is elevated to a whole new level that I don't think we have experienced before. Now, here's the part. I think society, Has yet to adapt. Even think about self-driving cars. We're still playing with the idea of self-driving cars. What about the self-driving doctor? A self-driving psychologist? You know, imagine this technology being able to be so intimate with you. It could access your phone, your computer, your phone, …

AI assessment note: “in a way it is a way you're trying to build as close to human intelligence”