The Exchanges

Q Do you, and do you believe in the humanoid form factor robots or do you think that's a little overkill?

A I'm probably the one weirdo who doesn't, who's not a big fan of humanoids, but maybe I'm jaded because we've, you know, we at least acquired at least two humanoid, uh, robotic startups and later sold them. Um, But, but the reason is, I mean, the reason people want to do humanoid robots for the most part is because the world is kind of designed around this form factor and, you know, you can train on YouTube. We can train on videos. People do all the things. Um, I personally don't think that's given the AI quite enough credit. Like AI can learn, you know, through simulation and through real life pretty quickly how to handle different situations and I don't know that you need exactly the same number of arms and legs and wheels, which is zero in the case of humans as humans to make it all work. And so I'm, I'm probably less Bullish on that, but to be fair, there are a lot of really smart people who are making humanoid robots, so I wouldn't discount it.

AI assessment note: “I'm probably the one weirdo who doesn't, who's not a big fan of humanoids”

Q foundational models, like if you look three years forward, Will they start to cleave off and get highly specialized? Like, beyond the general and the reasoning, maybe there's a very specific model for chip design. There's clearly a very specific model for biologic precursor design, protein folding. Like, is the number of foundational models in the future, Sergey, a multiple of what they are today? The same? Something in between?

A That's a great question. I kind of, If I, I mean, look, I don't know, like you guys could take a guess just as well as I can, but, um, if I had to guess, you know, things have been more converging. Uh, and, uh, this is sort of broadly true across machine learning. I mean, you used to have all kinds of different kinds of models and whatever, convolutional networks for vision things. And, you know, you had, um, What are our RNNs for text and speech and stuff? And, uh, you know, all of this has shifted to transformers basically, uh, and, uh, increasingly it's also just becoming one model. Um, now we do get a lot of boom. Occasionally we do specialized models, uh, and it's, it's definitely scientifically a good way to iterate when you have a particular target. You don't have to like do everything in every language and Handle whatever, both images and video and audio and, uh, in one go. Um, but we are generally able to, after we do that, take those learnings and basically put that capability into a general model. So there's not that much benefit. Um, you know, you could, you can get away with a somewhat smaller specialized model, a little bit faster, a little bit cheaper, but the trends have not gone that way.

AI assessment note: “if I had to guess, you know, things have been more converging.”

Q And what about the hardware? Like when you guys build stuff, do you care that you have this Pathway to NVIDIA? Or do you think eventually that'll get abstracted and there'll be a transpiler and it'll be NVIDIA plus 10 other options, so who cares? Let's just go as fast as possible.

A Well, we mostly, for, for Gemini, we mostly use our own TPUs. So, um, but we also do support, um, NVIDIA and we were one of the big, uh, uh, purchasers of NVIDIA chips and we have them in Google Cloud available for our customers, uh, in addition to TPUs. Um, at this stage, it's, uh, for better for us, not that abstract and maybe someday the AI will abstract it for us, but you know, given just the amount of computation you have to do on these models, you actually have to think pretty carefully how to do everything and exactly what kind of chip you have and how the memory works and the communication works and so forth, uh, are actually pretty big factors. And it actually, yeah, maybe one of these days the AI itself will be good enough to reason through that today. It's not quite good enough.

AI assessment note: “at this stage, it's, uh, for better for us, not that abstract”

Q And where have you seen the greatest success, surprising success in the application of models, whether it's in robotics or biology? What are you, like, seeing that you're like, wow, this is really working? And where are things gonna be more challenging and take longer than I think some people might be expecting?

A Um, Yeah, I mean, uh, now that you mention those. Well, I would say in biology, you know, we've had AlphaFold for quite a while, um, and I'm not personally a biologist, but when I talk to biologists out there, like, everybody uses it, and it's more recent, uh, variants, um, and that is, I guess, a different kind of AI, but like I said, I do think all these things tend to converge. Um, you know, robotics, For the most part, I see in this sort of wow stage, like wow, you could make a robot do that with just, you know, this general purpose language model, or just a little bit of fine tuning this way or that, and it's like amazing, um, but maybe not for the most part yet at the level of robustness that would make it like day to day useful.

AI assessment note: “in biology, you know, we've had AlphaFold... robotics... not for the most part yet”

Q So as you, as you have those moments, and then you go home to your just life as a dad, have you gotten to the point where you're like, what will my children do? And are they learning the right way? And should I totally just change everything that they're doing right now? Have you had any of those moments yet?

A Yeah. I mean, I look, I, I don't really know how to think about it to be perfectly honest. I don't have like a magical way. I mean, I see I've, uh, uh, kid in high school and middle school and, you know, I mean, the AIs are basically, you know, already ahead, you know, I mean, obviously there's some things AIs are particularly dumb at and they, you know, they make certain mistakes a human would never make, but generally, you know, if you talk about like math or calculus or whatever, like, They're pretty damn good. Like, they, you know, can win, like, math contests and coding contests, things like that against, you know, some top humans. And then I look at, you know, ok, he's, whatever, my son's gonna go on to, whatever, from sophomore to junior, and what is he gonna learn? And then I think in my mind, and I talked to him about this, well, what is the AI going to be in one more year?

AI assessment note: “Yeah. I mean, I look, I, I don't really know how to think about it”

Q Are there parts of the AI stack that interests you more than others right now? Are there certain problems that are just totally captivating you?

A Yeah, I started, uh, you know, like sort of, um, I don't know, a couple of years ago and maybe a year ago, uh, I was really very close with, uh, the, what we call pre-training. Yeah. Actually, most of what people think of as AI training, whatever people call it, pre-training, for various historical reasons, uh, but that's sort of the big, super, you know, you throw huge amounts of computers at it, um, and, uh, I, I learned a lot, you know, just being deeply involved in that, seeing us go from model to model, and so forth, and running little baby experiments, but, uh, kind of just for fun, so I Could say I did it. Uh, and more recently, the post-training, especially as the thinking models have come around. Um, and that's been, you know, another huge step up in general in AI. So, uh, you know, we don't really know what the ceiling is.

AI assessment note: “more recently, the post-training, especially as the thinking models have come around”