The Exchanges

Q and butter. So Before we get into the new architecture, just quickly, is the traditional way of training these LLMs, is that going to hit a wall? Because, you know, it seems like there's fairly little data left to use and the compute might be kind of running out also. So is it sort of like, not a nice to have, but a must have that new architectures are necessary?

A Um, I, ah, I actually think, ah, new architectures are absolutely necessary in the future, ah, and, ah, there are already hybrid architectures evolving. I mean, ah, I mean, um, in terms of state space models to actually connectors, uh, hybrid models between state space to, uh, transformers and so forth. Uh, but there are also actually how we are training some of these large language models with even the current data. How do they get reinforced and all, all those things. There is still a lot of room, uh, left in that comes to how these models are getting trained, uh, And, ah, you already are seeing actually even, ah, Meta, for instance, ah, published a book. It is all about actually high quality data than, ah, lots of data, ah, and then being able to cherry pick and do the right cleansing and so forth is number one. And then the second thing is also, it is not about bigger and bigger models when you look at actually what enterprises want, because your view is purely on just model providers who are trying to build models, but Take an enterprise, uh, who wants to use these models to actually build in, uh, chatbot agent or being able to, or market provider, like Rocket Mortgage, uh, and so forth, that they want to actually test it in production. It's not about actually picking the most capable model, but it, uh, in terms of model size or so forth, but they want to be able to take a…

AI assessment note: “I actually think, ah, new architectures are absolutely necessary in the future”

Q Well, I sort of disagree with you on that. I mean, I agree with you that the application is important, but the thing that's most important, of course, is how this stuff continues to evolve and get better, right? You wouldn't be satisfied just saying, let's stop with what we have now.

A No, actually, that is not what I'm saying. I'm actually saying, uh, that you're not hitting a wall from a customer perspective who are actually building or innovating with LLens because there is not enough data. I'm pointing out that even with these LLens, if I'm actually, uh, millions of developers who are trying to build amazing applications, what is important is taking these LLens and being able to customize it. And, uh, uh, that is the point. And, ah, and still there is so much innovation and value left to be done there. So that is the, ah, point I'm making. And because those set of data is not accessible to any of the model providers and so forth, and there is still a lot of, actually, innovation that is net new possible because, ah, those are actually in secure environments that are not accessible because most of these models are trained on public data or publicly available or

AI assessment note: “No, actually, that is not what I'm saying.”

Q Okay, great. I want to end with a little discussion about open source. Lama three is running through bedrock, I believe, or AWS day one. What is your view on open source compared to these other closed source models? Dangerous, good, helpful, just one of the, you know, one of the pack. What's your thoughts?

A Um, so, well, we are big supporters of, ah, actually, ah, enabling, ah, access to these models and, ah, ah, I mean, companies like, ah, Meta and, ah, Mistral have been Innovating at an incredible clip in some of these spaces and, ah, with the publicly available, ah, weights and so forth. So, ah, the key thing, and both of them are doing incredibly well, is also doing it in a thoughtful, responsible way to put the appropriate amount of guardrails and, ah, asphalt. So, ah, that's why I actually think, ah, this responsible innovation, ah, Even in open source is a good trend. And, uh, um, and I expect that to continue, especially with, uh, thoughtful innovation from some of these providers. And those are some of the reasons why we partner with them to even actually make these models accessible to all developers as well.

AI assessment note: “I actually think, ah, this responsible innovation, ah, Even in open source is a good trend.”

Q models evolve, can you talk a little bit about how we have actual, like the smaller models perform better than the last generation of bigger models? Is it, does, is it just like the training is different? You talked a little bit about cleaning the data that goes in there. I'm curious what would be involved in that, but how are these smaller models performing better than the bigger models?

A Um, there are a few, I mean, that notion of what is small and, uh, what is large, uh, we always tend to, uh, I mean, it's relative because our definition of small and large keeps changing, but equally important is there are multiple ingredients on what makes a great model. One is actually around, uh, how much compute, uh, it is, uh, we have thrown, but also how much data. That we are, ah, actually used to train these models, and how do they get reinforced, ah, so that the model knows actually it is delivering the right, ah, actually responses to certain questions or so forth. Ah, all these ingredients are right, and then how do you iterate based on some of these feedback, ah, ah, as well. These are some of the ingredients that make into, ah, how a model provider trains, ah, these models, and, ah, If you see, even if, ah, someone is actually training and, ah, much, ah, smaller model when it comes to parameters, but instead they actually have trained on huge number of tokens and so forth with better recipes on reinforcing, there is a case to be made, ah, that, ah, they are able to actually get, ah, higher quality results. Ah, the second, and if you see interesting research coming out of Stanford or others is, Also, when it comes to novel architectures, ah, that they are talking about, ah, that they expect to see also equally compelling, ah, models, ah, that are actually going to …

AI assessment note: “trained on huge number of tokens and so forth with better recipes on reinforcing”

Q our question, this is the point here. And then we're going to let you talk to our cash was like, didn't Amazon do this? And Mark Zuckerberg says you have to ask them. So I'm curious what you think about the constraints of data when it comes to training large language model models, and then maybe you can talk about your efforts since Mark Zuckerberg suggests that we ask you.

A All right. Uh, first of all, uh, just to give some context, I mean, uh, I'll just say AIS first, uh, has the most transformative technology and potential, uh, for us. It is the most transformative technology and, uh, The reason we are having this moment and all these, uh, conversation is because, uh, the advent of transformer technologies, which is, uh, the new neural net architecture, which helps us build these large language models that, uh, can learn from huge amount of data, but also be able to, uh, actually scale it up. That means you can throw more compute and more data and then it naturally gets better and better. So that is, uh, the reason why now suddenly AI is having its moment. Now, uh, to get to your question, where does it stop? Where is the, this is what researchers call it as, like, scaling laws. Saying, like, is it actually constantly are we going to keep throwing more and more compute and more and more, Data, or is there no more compute left, or no more data left in the world? And this is where a few things I suspect is going to happen. One, I don't think we are anywhere close to yet hitting that wall yet. So I do think we are going to actually see a lot of net new innovations when it comes to optimizing how to actually train these models in parallel and get better Utilization so that we can continue to actually build these models, ah, really well. I mean, AWS …

AI assessment note: “I don't think we are anywhere close to yet hitting that wall yet.”

Q little bit about your decisions about partnering versus building in house versus throwing other models out there. Um, Do you, I mean, do you, you have a large team building your own large language models in house at AWS, and you also have a very big investment in Anthropic. Can you talk through a little bit about what the logic is between kind of going at it in both ways?

A Yeah. I mean, uh, so, uh, if you see, uh, the history of AWS, we always believed, uh, in, uh, customer choice, uh, even when we actually started with, uh, and when I started the database team at that time, at that time, the world used to think databases actually like, uh, it has to speak SQL, and we kind of pioneered the art of, uh, hey, actually, customers, in fact, want different models, I mean, different databases for different use cases, and, uh, We ended up actually, uh, and the rest of the industry, um, and AWS now has the diverse set of portfolio on the database front, and rest of, uh, industry completely pivoted that way. And when internally, now going to the Gen AI world and, uh, so for internally, as we were building bunch of these Gen AI applications, uh, we saw what it takes to put together, uh, actually a model, uh, I mean, an end-to-end Gen AI application. Well, there is a notion of one powerful model that you need, uh, part of doing some other things. There are a bunch of different machine learning models, uh, that you need, uh, such as, um, you need a model to ingest your data, to do, like, uh, vectorize your data, and do retrieval augmented generation. You need a separate re-ranker model. You need, actually, a separate model for guardrails. You need an, uh, API orchestration to do software agent and automation, and so forth. And each of these models vary across…

AI assessment note: “This led to our belief that no single model will be ideal for every use case”