The Exchanges

Q in that this market would be really important, and you more than others, right, since you actually started a company in it. But then it took some time for the market to really expand to the point where, uh, to your point now, it's, it's this massive use case. People really care about speed of inference and other things. Um, what gave you the conviction back then to do this?

A Combination of, of vision, um, the right co-founders, and a little bit of arrogance, a little bit of luck. You know, we, we saw AI on the horizon as a new workload. And as computer architects, new workloads are opportunity, right? It's very, very hard to, to, to enter in the x-eighty-six world, right? Where there's not, nothing new is happening there, and nothing has happened for generations. But You know, when graphics emerged, you got the discrete GPU and you, you, you got, uh, Nvidia and, and when, uh, when the mobile, uh, compute hit, you, you got arm. And it was interesting that, that not Intel, not AMD, not all sorts of people who you would have thought have been really well positioned to win in that business. They all got no share. And so we knew that, that this new workload would eat a lot of compute. It would require Uh, a new architecture, dedicated architecture, and that ought to be very different. The architecture could not be a derivative of what's existing. Those were our big bets, and they were a hundred percent contrarian, and they turned out to be dead right.

AI assessment note: “Combination of, of vision, um, the right co-founders, and a little bit of arrogance”

Q Can I ask you as an aside, actually, just because you, you have for more than a decade believe that this revolution is going to happen. Uh, how much is all of this, um, AI generated coding relevant for Cerebris internally?

A Hugely. I would say that, that, you know, eight months ago, we weren't spending a thousand dollars in engineer on tokens, and we're probably at 25 or 30,000 right now, and it's ripping. I, I think it's not useful for everybody. I, I think that's the truth. I, I think there are some, some people who have sort of the perfect mindset for it, right? And you, you, they are running eight or 10 agents, seven by 24. They've moved their coding, Style to being one in which they govern agents, whether they think about how to QA, so they've got a QA agent running, they think about how to sort of remedy some of the weaknesses in the coding models, right, they're often verbose, they often cut out comments, so they've really thought about, and it's a type of puzzle that's the perfect fit for their mind, and they've gone from being sort of 10 X guys to being hundred X guys. I think the rest of us, myself included, we're sort of Limping along. We, we, we're trying to figure out how, how we can make it work for, for our different jobs, for being the CEO, for being the CFO, for being accountants, for being in marketing. Um, but for a, a small number, it is such a tool. And then the rest, we try and, try and show them what, what, what others are doing, what best practices are.

AI assessment note: “Hugely. I would say that, that, you know, eight months ago, we weren't spending”

Q Why did you guys get into the, um, training and open source model game?

A We had so much compute and that it, it was a way we could prove to the world how easy it was to train on us. We felt it was evidence that we could build and our systems could train. The, the biggest and fastest models, the most accurate models in the space. In March, we put seven GPT models in the open source community. Everybody else was putting one. Why? Because it's really hard to redistribute work across a GPU cluster. For us, it's one keystroke. So we put seven. People are coming to us with extraordinary ideas. We had a customer who came and they said, look, we'd like you to, to design for us a, a model. At three billion that we could prune and quantize such that it could be served off a cell phone. Really cool.

AI assessment note: “it was a way we could prove to the world how easy it was”

Q performant open source models, and you mentioned that as you scale up a model, inference cost starts to really kick in as you get to bigger and bigger models from a parameter perspective. What do you view as the limits to scaling these models? Like at what point do they get too big, or do you think we just keep scaling them until we have these sort of hyper models?

A It's a really interesting optimization problem. They get big, they get harder to work with. It's harder to retrain, right? What even GPT-IV came out Didn't have any insight beyond twenty-twenty-one. They had to race. That was because it was so big, right? And, and so I think there are trade-offs. There are trade-offs between accuracy and size. There are trade-offs between size and cost to do inference. I mean, maybe I want a larger model for radiology files, right? Or for my doctors. Maybe I'll take a smaller model for my, for my chat or, or my customer service bot, right? We're going to have to think about these in, in terms of what they're delivering, what the cost of being wrong is, what the cost of serving is, and, and, and think about this as a, as a business decision. And at first, everyone's just running, trying to say, I can make a bigger model. I can make a bigger model. And then the guys who, Trying to run businesses, you're like, well, I can't afford to give that away. And so we're going to be down at three and at six billion and at thirteen billion, because that's, I can get pretty good, pretty darn good, and not break the bank with free inference. And so I, I think that there are these trade-offs that are, are really challenging, and we're just beginning to grapple with now.

AI assessment note: “There are trade-offs between size and cost to do inference.”

Q Andrew, you have such a unique expertise and depth and a question that everybody is thinking about right now. Can you give our listeners an overview of, um, the AI accelerator market right now? Like who are the big buyers and then how might they decide to do anything but NVIDIA?

A NVIDIA's made hay when the sun shines and they've done an extraordinary job and you have to take your hat off to them, right? I think they're now in a situation where they're extorting customers. They're extremely expensive. They're unable to ship. And that has, among other things, opened the door for many of us who, who, who have alternatives. Nobody likes being dependent on their vendor. You can ask the guys at Google and Facebook who are dependent on Intel for years, how much they hated that. They dislike that intensely. And so I, I think there's a battle between people's sort of dislike being dependent and the need just to keep running forward. I think large enterprises continue to be a good part of our business. I think often where you can provide a little consulting to help them accelerate their model deployment is something that provides an opportunity for, for smaller companies to get in the door. And when you show them how, how much easier it is, we had a situation where they were trying to train on a GPU cluster and they were at 60 days and it wasn't converging and We stood it up, and three and a half days later, their model converged, and they were like, holy cow. We have an internal cloud, so customers can just jump on your cloud. They can begin training right away. If they want to try before they buy, they can begin with a little training run and go from there. We …

AI assessment note: “opened the door for many of us who, who, who have alternatives.”

Q Can you talk a little bit about, I'm so curious, like, how did the OpenAI deal happen? You know, what were, um, what do you think was the point at which you, you knew that you were a good fit for them?

A I, I think I spoke to Sam in, in sort of middle of the summer in, in 25, and he said for the first time, he, he said, we're, we've been trying so hard just to keep up with demand. We, we now see the importance of fast inference. That produced a set of trials and some testing that, that was done. Um, and we were so much faster than the, than the competition. It felt really good. And when we love talking to super smart customers, right? I mean, I, I can't, I know you do consumer too. I, I can't do consumer. I have a rule that if my, my mother buys it or uses it, I don't want to make it or sell it. Um, cause I, I, I really want super smart customers who are doing really interesting things with our stuff. And so we got in with, um, some of their guys and they were like, whoa, This is, we understand now. And at Thanksgiving, the night before Thanksgiving, we signed a term sheet. And, you know, four weeks later, on the 24th of December, we signed a, a big master agreement. And so, um.

AI assessment note: “at Thanksgiving, the night before Thanksgiving, we signed a term sheet.”