The Exchanges

Q I, I totally get you there. Can I ask in terms of like, you know, the transition between models, implementation is not that easy. You're not just handing over data with ease. What are the biggest challenges to adoption do you think for startups and for companies moving forwards when they think about working with new models and kind of data migration to them?

A There are a lot of issues. Um, probably the, the most obvious one is around the Correctness and dependability of answers. If you ask folks, one of the key concerns is like, well, these models make up stuff. So that, that is the obvious one. But then there are second order issues, important, but maybe less obvious for, for people, which is security and privacy of data, the data that is used for the question. And then there are even more complex questions on the rights to The answers. The Wall Street company asked me if we feed a number of portfolio trading strategies into a model, and it makes a recommendation, and the recommendation makes money. Could anyone have claims on that, um, answer? And it gets very complicated very quickly.

AI assessment note: “most obvious one is around the Correctness and dependability of answers”

Q Do you not think there's like an inherent challenge just in terms of the opacity of models? Like when we think about the regulatory challenges, We can't continue to have such opacity to get to outcomes. Will there not need to be more transparency in models to show people the pathway to answers?

A So a hundred percent, um, two examples. One again from last week, IBM was announcing their own Gen AI models, and they were talking about being fully transparent on the data that went into the models. That's a big step in, in, in lineage. The other one is, is talking about how they trained it. So they're, they're, they're in tune with this concern and they want to, to surface it, uh, in a more, uh, open way. The other example is at Snowflake, we acquired this company Neva that was doing consumer search based or augmented by LLMs. And a lot of the principles that they had when they created this product was you want to have Citations and quotations and attribution, even if it's a summary, every snippet, what was the source that it came from, which is a way to, um, control for hallucination. So I would say that that is the world that we're headed towards, at least in the enterprise, like forget the creative side of things, but at least in the, in the world where you need correct answers, you need to be able to attribute where things came from.

AI assessment note: “So a hundred percent, um, two examples.”

Q A really hard one. That's just a subsequent one from that. I had Gustav, who's the CPO at Spotify on the show, and he says, talk is cheap. So we should do more of it. How do you think about the balance between internal debate on product and product ideas, iterations versus just speed of execution and getting it done?

A I think it depends on the nature of the technology or the nature of the product. At Snowflake, we have both types of technology. So the core subsystem that does say clustering of data on disk I think you want to design that thing really, really well. Measure a hundred times and cut once because nobody wants their data to get corrupted or their results to be wrong if that thing is not built the right way. But if you want the, the UI for a query editor and you have 10 different ways on how you could do suggestions for customers, there's no right and wrong. Might as well go quickly, iterate, learn from, from users. So I would say both are the right tools and the right approaches depending on what you're trying to do.

AI assessment note: “I think it depends on the nature of the technology or the nature of the product.”

Q A really hard one. That's just a subsequent one from that. I had Gustav, who's the CPO at Spotify on the show, and he says, talk is cheap. So we should do more of it. How do you think about the balance between internal debate on product and product ideas, iterations versus just speed of execution and getting it done?

A I think it depends on the nature of the technology or the nature of the product. At Snowflake, we have both types of technology. So the core subsystem that does say clustering of data on disk I think you want to design that thing really, really well. Measure a hundred times and cut once because nobody wants their data to get corrupted or their results to be wrong if that thing is not built the right way. But if you want the, the UI for a query editor and you have 10 different ways on how you could do suggestions for customers, there's no right and wrong. Might as well go quickly, iterate, learn from, from users. So I would say both are the right tools and the right approaches depending on what you're trying to do.

AI assessment note: “I think it depends on the nature of the technology or the nature of the product.”

Q Do you not think there's like an inherent challenge just in terms of the opacity of models? Like when we think about the regulatory challenges, We can't continue to have such opacity to get to outcomes. Will there not need to be more transparency in models to show people the pathway to answers?

A So a hundred percent, um, two examples. One again from last week, IBM was announcing their own Gen AI models, and they were talking about being fully transparent on the data that went into the models. That's a big step in, in, in lineage. The other one is, is talking about how they trained it. So they're, they're, they're in tune with this concern and they want to, to surface it, uh, in a more, uh, open way. The other example is at Snowflake, we acquired this company Neva that was doing consumer search based or augmented by LLMs. And a lot of the principles that they had when they created this product was you want to have Citations and quotations and attribution, even if it's a summary, every snippet, what was the source that it came from, which is a way to, um, control for hallucination. So I would say that that is the world that we're headed towards, at least in the enterprise, like forget the creative side of things, but at least in the, in the world where you need correct answers, you need to be able to attribute where things came from.

AI assessment note: “So a hundred percent, um, two examples.”

Q respectfully, the startups that maybe didn't go to plan, we could say, when you think about Google and Microsoft, they're such symbolic institutions in our environment. What are one to two of your biggest takeaways from 13 years at Microsoft? I mean, shit, that's a long time. And then, you know, the four to five years that you had at Google, what are one or two of those big takeaways?

A Yeah, from, from Microsoft, I think I got the, The ease of use and value of simplicity in products. At the time, SQL Server was coming from way behind competing with market leaders that were at Oracle and IBM, and the way in was not to have every single feature and capability that those technologies had. It was just simplify things. If you turn something that is a subset of the capability, but dramatically easier to use, That gets a following, and that was a very, very clear lesson learned, and I've seen it over and over, and by the way, the Snowflake story follows a big part of that journey, which is if you simplify things to a point that it is delightful to use, People adopt. That's from the Microsoft time. Let me think from the YouTube time, maybe the biggest lesson learned is that consumer products have many more elements beyond just technical difficulty. There's a lot of timing. What are the trends with consumer behavior? Um, sure. You need to have a good business model. You need to have the technology, but there are, there are things that You can control.

AI assessment note: “from Microsoft, I think I got the, The ease of use and value of simplicity”

Q all get a little bit overexcited about, which is like speed of adoption always takes longer than people think. Many, if you can believe it, Christian enterprises in Europe still have no idea what Slack is. And so my question to you is when we think about the different verticals, who do you think is first to adopt and the fast movers? Who do you think is slow to adopt?

A I think it completely correlates with data maturity. You may have heard some of us at Snowflake talk about There's no AI or gen AI strategy without a data strategy. It's not just a line. It's a truth that we strongly believe in. And from that perspective, I would say financial services are at the forefront. Most of the financials have figured out for a long time how to organize data and leverage data for competitive advantages. Go look at the sophistication of hedge funds as an example. Retail and CPG companies have also Been very wise at using data. Maybe I would point that public sector, not that they're not data savvy, but they have a lot of regulation and constraints that may make it harder for them to adopt.

AI assessment note: “I would say financial services are at the forefront... public sector, not that they're not”

Q They absolutely are in many ways research companies. The thing I want to ask is like often the size of the model is quite hailed. How important do you think model size is today, Christian?

A I think it depends on the use case. For a Generic consumer product like ChatGPT, where anything is fair game. The model is supposed to know about every possible topic. It speaks every language, et cetera. I would say that for those use cases, a model that is large and has lots of cumulative knowledge is very valuable. I would say that for specialized use cases, which is what I see more in the enterprise model matters less. If anything, model size will influence things like cost and latency. So smaller may be better. And now there's plenty of examples that have been run where a smaller model fine tune where a specific purpose or a specific data set produces results better than a generic model. So I would say it's use case dependent.

AI assessment note: “I think it depends on the use case.”

Q Christian, I am so excited for this. I've heard so many good things. So I would love to start with Your entry into product. How did you come to be SVP of product at Snowflake? Let's start there, Christian.

A Uh, thank you for having me, Harry. Um, background. Born and raised in Colombia and South America. I did a startup there. I learned what not to do. I did another startup in the U.S. I learned what else not to do. So at some point, like, I need to learn from, from the guys that really know how to build software. This is 1999. I joined Microsoft. Did a long stint in data all the time. SQL server, appliances when appliances were the thing to build and then cloud services. And from then I went over to YouTube at Google where I was responsible for the infrastructure, including data systems. And I think all of that set me up for understanding data, being a data junkie. And when the opportunity opened up for Snowflake, I'm like, I could appreciate the technology and the type of company. So I'm like, I'm ready to be there.

AI assessment note: “when the opportunity opened up for Snowflake, I'm like, I could appreciate the technology”

Q What do you think is most exciting? When you look at the different verticals, use cases, opportunities, what do you think is most exciting?

A I would say that this is a real shot in the arm to the creative businesses. And that's, I think where, where we talked about generative AI, that it's generating or creating something. That's where we saw the initial use cases and stable diffusion or mid journey and those things in many ways, because what in. Other industries we, we would call a bug or a hallucination. In the creative world, there are features. They're, they're, they're, so it's a goodness to come up with something that has not been done before, or it's a mix and match of existing things. So creative industries are probably the sweetest spot. I love what Adobe has been doing with their products and, and how they've integrated Gen AI. Massive kudos to them. Then there's the opportunity in every other industry, in every other vertical. It's only that that requires with, it requires a little bit of Understanding correctness, understanding data maturity, things like that, but every customer that I talk to these days, they're looking into doing something. They're all trying to figure out how to get started and how to get there, but I think it applies to all sorts of businesses.

AI assessment note: “creative industries are probably the sweetest spot”

Q a rapper on top of a, you know, GPT model, like, And it's kind of brushed off in that way. It's just a GPT wrapper. Do you think that's fair? And actually these models and the providers will create and kill off all the startups with their verticalized use cases? Or do you actually think that VCs are being short-sighted with the kind of handoff of it's just a wrapper?

A I think there's many categories. There are some very shallow wrappers on top of a GPT-IV. I don't place much value on them. I usually ask, hey, how long did it take you to build this? Oftentimes it's a week or two. I don't think there's a company there. I do think that there are some very deep wrappers on top of GPT-IV that apply domain specific legal or other domain that I think you end up with a true way to bring GPT-IV to a given market or industry. I think those are value. There are actually many startups chasing the different aspects of, um, innovation on the core technology. I, I was at a dinner a few weeks ago and someone was saying, we're trying to blend fine tuning with prompting. Someone else was saying, we're trying to address the limitations of the transformer model. Someone else was trying to look at, um, how to, how to do computer vision better. Once you start looking at it, Core tech, not just application of existing models. I think there's innovation everywhere, which is why I would say there will be a number of new startups creating new things and probably the opening eyes and Anthropics are continuing their innovation, which those are effectively research companies.

AI assessment note: “there will be a number of new startups creating new things”

Q It was the charm and charisma of Frank. I don't blame you. I, I had the same feeling when he looked into my eyes. I do have to ask, you mentioned that you learned what not to do. If there were one or two things that you really learned what not to do, what would they be, Christian?

A I would say talent being the driver of truly great outcomes. I would say don't ever compromise on talent. Don't ever say, yeah, this person doesn't have a background, but maybe Has the right intention. Just take a bet. No, I think that nothing substitutes talent. That's a very clear lesson learned. And, and the other thing that has been very clear is building a scalable business is difficult. Well, one of those startups, we, we built some scheduling software for airlines and the thesis was you build it once works, then you just resell it and you can be the next Microsoft. It was not the case. There was a lot of customization. It would turn out into more of a services business. So scalability and building platforms was another lesson learned.

AI assessment note: “I would say don't ever compromise on talent.”

Q all get a little bit overexcited about, which is like speed of adoption always takes longer than people think. Many, if you can believe it, Christian enterprises in Europe still have no idea what Slack is. And so my question to you is when we think about the different verticals, who do you think is first to adopt and the fast movers? Who do you think is slow to adopt?

A I think it completely correlates with data maturity. You may have heard some of us at Snowflake talk about There's no AI or gen AI strategy without a data strategy. It's not just a line. It's a truth that we strongly believe in. And from that perspective, I would say financial services are at the forefront. Most of the financials have figured out for a long time how to organize data and leverage data for competitive advantages. Go look at the sophistication of hedge funds as an example. Retail and CPG companies have also Been very wise at using data. Maybe I would point that public sector, not that they're not data savvy, but they have a lot of regulation and constraints that may make it harder for them to adopt.

AI assessment note: “I would say financial services are at the forefront”

Q They absolutely are in many ways research companies. The thing I want to ask is like often the size of the model is quite hailed. How important do you think model size is today, Christian?

A I think it depends on the use case. For a Generic consumer product like ChatGPT, where anything is fair game. The model is supposed to know about every possible topic. It speaks every language, et cetera. I would say that for those use cases, a model that is large and has lots of cumulative knowledge is very valuable. I would say that for specialized use cases, which is what I see more in the enterprise model matters less. If anything, model size will influence things like cost and latency. So smaller may be better. And now there's plenty of examples that have been run where a smaller model fine tune where a specific purpose or a specific data set produces results better than a generic model. So I would say it's use case dependent.

AI assessment note: “I think it depends on the use case.”

Q all like, yeah, so it's exciting. How do, how do I do that, Harry? And I think the biggest businesses in AI will be built in the implementation services businesses, helping large enterprises implement AI in a meaningful way into their enterprise over the next decade. Do you agree with me in terms of this lack of enterprise education on implementation and the potential opportunity for implementation services given that?

A I agree part of it is, ah, services, implementation, education, but I also think that the stack and the way to think about this is evolving. I don't think that we have a very clean This is the, the three or four components that you use, and this is the type of use case that you apply. Companies are playing with, hey, the LLM is magical, and you send it, send it to questions, and magic answers come back. Some others have combined, uh, vector retrieval with LLMs. But within that, there's a lot of variability. Which model, which database or which vector database, how much do you prompt versus fine tune? So I would say a lot of that is still being figured out. And I think the, the stack continues to evolve, will continue to mature and in parallel, okay, then company need to get educated on when to use what for what use case.

AI assessment note: “I agree part of it is, ah, services, implementation, education, but I also think”

Q respectfully, the startups that maybe didn't go to plan, we could say, when you think about Google and Microsoft, they're such symbolic institutions in our environment. What are one to two of your biggest takeaways from 13 years at Microsoft? I mean, shit, that's a long time. And then, you know, the four to five years that you had at Google, what are one or two of those big takeaways?

A Yeah, from, from Microsoft, I think I got the, The ease of use and value of simplicity in products. At the time, SQL Server was coming from way behind competing with market leaders that were at Oracle and IBM, and the way in was not to have every single feature and capability that those technologies had. It was just simplify things. If you turn something that is a subset of the capability, but dramatically easier to use, That gets a following, and that was a very, very clear lesson learned, and I've seen it over and over, and by the way, the Snowflake story follows a big part of that journey, which is if you simplify things to a point that it is delightful to use, People adopt. That's from the Microsoft time. Let me think from the YouTube time, maybe the biggest lesson learned is that consumer products have many more elements beyond just technical difficulty. There's a lot of timing. What are the trends with consumer behavior? Um, sure. You need to have a good business model. You need to have the technology, but there are, there are things that You can control.

AI assessment note: “from Microsoft, I think I got the, The ease of use and value of simplicity”

Q all like, yeah, so it's exciting. How do, how do I do that, Harry? And I think the biggest businesses in AI will be built in the implementation services businesses, helping large enterprises implement AI in a meaningful way into their enterprise over the next decade. Do you agree with me in terms of this lack of enterprise education on implementation and the potential opportunity for implementation services given that?

A I agree part of it is, ah, services, implementation, education, but I also think that the stack and the way to think about this is evolving. I don't think that we have a very clean This is the, the three or four components that you use, and this is the type of use case that you apply. Companies are playing with, hey, the LLM is magical, and you send it, send it to questions, and magic answers come back. Some others have combined, uh, vector retrieval with LLMs. But within that, there's a lot of variability. Which model, which database or which vector database, how much do you prompt versus fine tune? So I would say a lot of that is still being figured out. And I think the, the stack continues to evolve, will continue to mature and in parallel, okay, then company need to get educated on when to use what for what use case.

AI assessment note: “I agree part of it is, ah, services, implementation, education, but”

Q a rapper on top of a, you know, GPT model, like, And it's kind of brushed off in that way. It's just a GPT wrapper. Do you think that's fair? And actually these models and the providers will create and kill off all the startups with their verticalized use cases? Or do you actually think that VCs are being short-sighted with the kind of handoff of it's just a wrapper?

A I think there's many categories. There are some very shallow wrappers on top of a GPT-IV. I don't place much value on them. I usually ask, hey, how long did it take you to build this? Oftentimes it's a week or two. I don't think there's a company there. I do think that there are some very deep wrappers on top of GPT-IV that apply domain specific legal or other domain that I think you end up with a true way to bring GPT-IV to a given market or industry. I think those are value. There are actually many startups chasing the different aspects of, um, innovation on the core technology. I, I was at a dinner a few weeks ago and someone was saying, we're trying to blend fine tuning with prompting. Someone else was saying, we're trying to address the limitations of the transformer model. Someone else was trying to look at, um, how to, how to do computer vision better. Once you start looking at it, Core tech, not just application of existing models. I think there's innovation everywhere, which is why I would say there will be a number of new startups creating new things and probably the opening eyes and Anthropics are continuing their innovation, which those are effectively research companies.

AI assessment note: “There are some very shallow wrappers... I do think that there are some very deep wrappers”

Q What do you think is most exciting? When you look at the different verticals, use cases, opportunities, what do you think is most exciting?

A I would say that this is a real shot in the arm to the creative businesses. And that's, I think where, where we talked about generative AI, that it's generating or creating something. That's where we saw the initial use cases and stable diffusion or mid journey and those things in many ways, because what in. Other industries we, we would call a bug or a hallucination. In the creative world, there are features. They're, they're, they're, so it's a goodness to come up with something that has not been done before, or it's a mix and match of existing things. So creative industries are probably the sweetest spot. I love what Adobe has been doing with their products and, and how they've integrated Gen AI. Massive kudos to them. Then there's the opportunity in every other industry, in every other vertical. It's only that that requires with, it requires a little bit of Understanding correctness, understanding data maturity, things like that, but every customer that I talk to these days, they're looking into doing something. They're all trying to figure out how to get started and how to get there, but I think it applies to all sorts of businesses.

AI assessment note: “I would say that this is a real shot in the arm to the creative businesses.”

Q You said there about kind of, there's no gen AI without being kind of an incredible data strategy. You said to me before, generative AI is democratizing data access. You left me with that as a cliffhanger, Christian. So how so, and why do you believe this?

A Yeah. If you think of the role of traditional business intelligence technology, what's to sort of bridge the impedance mismatch between the business and business terminology and business users, a technology that Frankly, it's just for a few people, like writing SQL statements is not something that most people in a company do, and that was the role of that technology to do, to do that translation, but it still requires some mapping and curation, and effectively, how do you inform that impedance mismatch? I think Gen AI has the opportunity to Turbo charge this type of translation where the language is natural language, and the answers come in natural language, but along the way, there's traditional database lookups, traditional retrieval, and has the opportunity to democratize data, uh, dramatically more than where we are today. Another, another BI has not done a really good job, but I think it's, it's going to be now data for everyone.

AI assessment note: “Gen AI has the opportunity to Turbo charge this type of translation where the language is natural language”

Q Why are models as little as 10? And bluntly, why is such value being placed on the likes of OpenAI, Bard, Anthropic, if actually models are 10 and not A significant chunk more.

A Well, I would say if you look at it where, where things are today or where they were six months ago, maybe models would have taken, uh, a bigger edge because they paved the way on how do you model the data? You could make the case that has been there all along, but I would stick to the 10% or the smaller number because now you see how many foundation models are being created. We've seen companies that with Seven employees have creating models that are comparable for some use cases to what OpenAI or Anthropic do. So I would say at the end of the day, it's a data problem and model. These are strong words, but I think they're getting commoditized until the next big innovation comes and you allocate some more value to the model.

AI assessment note: “I think they're getting commoditized until the next big innovation comes”

Q Why are models as little as 10? And bluntly, why is such value being placed on the likes of OpenAI, Bard, Anthropic, if actually models are 10 and not A significant chunk more.

A Well, I would say if you look at it where, where things are today or where they were six months ago, maybe models would have taken, uh, a bigger edge because they paved the way on how do you model the data? You could make the case that has been there all along, but I would stick to the 10% or the smaller number because now you see how many foundation models are being created. We've seen companies that with Seven employees have creating models that are comparable for some use cases to what OpenAI or Anthropic do. So I would say at the end of the day, it's a data problem and model. These are strong words, but I think they're getting commoditized until the next big innovation comes and you allocate some more value to the model.

AI assessment note: “I think they're getting commoditized until the next big innovation comes”

Q about kind of data as a competitive mode, you know, I've had a lot of people on the show say before, bluntly, that data is so freely accessible today. It's no longer this prized possession that incumbents can hail and use to their advantage given how freely accessible it is. To what extent do you still place a premium on data ownership to leverage versus the freedom to access data?

A Yeah, I would separate there's both public data and private data, privately owned by enterprises. But the premise of your question is based on public data. And I would say If things did not change, the language models, or the models in general, would all converge towards, they're all training on the same data, and at some point it's what mix of data you use, but you'll trend towards the same answer. The interesting trend towards there is the notion of many companies realizing that their data is being used and monetized by these models, so there are Companies rethinking and changing their data policies. Are you allowed to crawl me? Are you allowed to train models with this? I think all of this will shift. In the next six, 12 months, this is all starting already, because in the same way that search changed the rules of engagement with public data, Gen AI is doing the same thing, and the companies that are behind that data are doing so.

AI assessment note: “I would separate there's both public data and private data, privately owned by enterprises.”

Q You said there about kind of, there's no gen AI without being kind of an incredible data strategy. You said to me before, generative AI is democratizing data access. You left me with that as a cliffhanger, Christian. So how so, and why do you believe this?

A Yeah. If you think of the role of traditional business intelligence technology, what's to sort of bridge the impedance mismatch between the business and business terminology and business users, a technology that Frankly, it's just for a few people, like writing SQL statements is not something that most people in a company do, and that was the role of that technology to do, to do that translation, but it still requires some mapping and curation, and effectively, how do you inform that impedance mismatch? I think Gen AI has the opportunity to Turbo charge this type of translation where the language is natural language, and the answers come in natural language, but along the way, there's traditional database lookups, traditional retrieval, and has the opportunity to democratize data, uh, dramatically more than where we are today. Another, another BI has not done a really good job, but I think it's, it's going to be now data for everyone.

AI assessment note: “Turbo charge this type of translation where the language is natural language”

Q about kind of data as a competitive mode, you know, I've had a lot of people on the show say before, bluntly, that data is so freely accessible today. It's no longer this prized possession that incumbents can hail and use to their advantage given how freely accessible it is. To what extent do you still place a premium on data ownership to leverage versus the freedom to access data?

A Yeah, I would separate there's both public data and private data, privately owned by enterprises. But the premise of your question is based on public data. And I would say If things did not change, the language models, or the models in general, would all converge towards, they're all training on the same data, and at some point it's what mix of data you use, but you'll trend towards the same answer. The interesting trend towards there is the notion of many companies realizing that their data is being used and monetized by these models, so there are Companies rethinking and changing their data policies. Are you allowed to crawl me? Are you allowed to train models with this? I think all of this will shift. In the next six, 12 months, this is all starting already, because in the same way that search changed the rules of engagement with public data, Gen AI is doing the same thing, and the companies that are behind that data are doing so.

AI assessment note: “there are Companies rethinking and changing their data policies.”