The Exchanges

Q much different for an LLM because you can expect a person to look at maybe the Five, six links in a Google search versus for an LLM, should you expect to have 20 links that are highly relevant? Like, how do you internally figure out, you know, how do we build the AI mode that is like maybe like much broader search and span versus like the more human one?

A Yeah, I mean, I think even pre-language model based work, you know, our ranking systems would be built to start with a giant number of web pages in our index. Many of them are not relevant. So you identify a subset of them that are relevant with very lightweight kinds of methods. You know, you're down to like, 30,000 documents or something. And then you gradually refine that to apply more and more sophisticated algorithms and more and more sophisticated Sort of signals of various kinds in order to get down to ultimately what you show, which is, you know, the final 10 results or, you know, 10 results plus other kinds of information. And I think an LLM based system is not going to be that dissimilar, right? You're going to tend to trillions of tokens, but you're going to want to identify, you know, what are the 30,000 ish documents that are with the, you know, uh, maybe Thirty million interesting tokens. And then how do you go from that into what are the 117 documents I really should be paying attention to in order to carry out the tasks that the user has asked me to do? Um, and I think, you know, you can, you can imagine systems where you have, you know, a lot of, uh, highly parallel processing to identify those initial 30,000 candidates, maybe with very lightweight kinds of models. Um, then you have some system that sort of helps you narrow down from 30,000 to the 117, uh, with…

AI assessment note: “an LLM based system is not going to be that dissimilar”

Q as like potential retrieval artifacts for the model? Like, or do I have like the edge cases, it's like a good example, right? It's like, If you're building systems, you already have in your mind specific edge cases depending on it, but now you have to, like, every time repeat it. Like, are you having people spend a lot more time writing out more generic things to bring back or?

A Um, I mean, I do think well-written guides of, of how to do good software engineering are going to be useful because they can be used as input to models or, you know, read by other developers so that their prompts are You know, more clear about what the underlying software system should, should be doing. Um, you know, I think It may not be that you need to create a custom one for every situation. If you have general guides and put those into, you know, the context of a coding agent, that, that can be helpful. Like in, you can imagine one for distributed systems. You could say, okay, think about failures of these kinds of things. And these are some techniques you can deal with failures. You know, you can have, uh, you know, Paxos like replication, or, you know, you can, uh, Send the request to two places and tolerate failure because you only need one of them to come back. You know, a little description of 20 techniques like that in building distributed systems probably would go a long way to having a coding agent be able to sort of cobble up more reliable and robust distributed systems.

AI assessment note: “well-written guides of, of how to do good software engineering are going to be useful”

Q of like, is there a way to bring the thousand picojoules down to 50? Like, is it worth designing a new chip to do that? The extreme is like when people say, oh, you should burn the model on the ASIC, and that's kind of like the most extreme thing. How much of it is it worth doing in hardware when things change so quickly? Like, what's the internal discussion?

A Yeah, I mean, we, we have a lot of interaction between, say, the TPU chip design architecture team and the sort of higher level modeling, uh, experts because we really want to take advantage of being able to co-design what should future TPUs look like based on where we think the sort of ML research puck is going, uh, in some sense because, uh, you know, as a hardware designer for ML in particular, you're trying to design a chip starting today And that design might take two years before it even lands in a data center, and then it has to sort of be a reasonable lifetime of the chip to take you three, four, or five years. So you're trying to predict two to six years out where, what ML computations will people want to run two to six years out in a very fast changing field. And so having people with interesting ML research ideas Of things we think will start to work in that time frame, or will be more important in that time frame, uh, really enables us to then get, you know, interesting hardware features put into, you know, TPU N plus two, where TPU N is what we have today.

AI assessment note: “we really want to take advantage of being able to co-design what should future TPUs look like”

Q And so like, how do we, I guess, do we want to extract that? Can we, can we divorce knowledge from reasoning, you know?

A Yeah, I mean, I think you do want the model to be most effective at reasoning if it can retrieve things, right? Because having the model devote precious parameter space to remembering obscure facts that could be looked up is actually not the best use of that parameter space, right? Like you might prefer something that is more generally useful in more settings than this obscure fact that it has. Um, so I think that's always a tension. At the same time, you also don't want Your model to be kind of completely detached from, you know, knowing stuff about the world, right? Like it's probably useful to know how long the Golden Gate Bridge is just as a general sense of like how long are bridges, right? And, uh, it should have that kind of knowledge. It maybe doesn't need to know how long some teeny little bridge in some other more obscure part of the world is, but, uh, It does help it to have a fair bit of world knowledge, and the bigger your model is, the more you can have. But I do think combining retrieval with sort of reasoning and making the model really good at doing multiple stages of retrieval and reasoning through the intermediate retrieval results is going to be a pretty effective way of making the model seem much more capable. Because if you think about, say, a personal Gemini,

AI assessment note: “having the model devote precious parameter space to remembering obscure facts that could be looked”

Q the cycle of these ideas, even like, you know, sparse models and, uh, you know, how, how do you reevaluate them? How do you think about in the next generation of model, what is worth revisiting? Like, uh, yeah, they're just kind of like a, you know, you worked on so many ideas that end up being influential, but like in the moment they might not feel That way, necessarily.

A Yeah, I mean, I think distillation was originally motivated because we were seeing that we had a very large image data set at the time, you know, three hundred million images that we could train on with, you know, I forget, like 20,000 categories or something, so much bigger than ImageNet. And we were seeing that if you create specialists for different subsets of those image categories, you know, this one's going to be really good at sort of mammals, and this one's going to be really good at Sort of indoor room scenes or whatever, and you can cluster those categories and train on an enriched stream of data after you do pre-training on a much broader set of images. You get much better performance if you then treat that whole set of maybe 50 models you've trained as a large ensemble. Um, but that's not a very practical thing to serve, right? So distillation really came about from the idea of, okay, what if we want to actually serve that and train all these independent sort of expert models, um, and then squish it into something that actually fits in a form factor that you can actually serve. And that's, you know, not that different from what we're doing today. You know, often today we're, instead of having an ensemble of 50 models, we're having a much larger scale model That we then distill into a much smaller scale model.

AI assessment note: “distillation really came about from the idea of, okay, what if we want to actually serve that”

Q Um, why Netherlands by the way? Or is it, is that because of Chrome?

A Uh, we had a data center in there. Um, so, I mean, I think this gets to the point of being able to do the back of the envelope calculations. So these are sort of the raw ingredients of those, and you can use them to say, okay, well, if I need to design a system to do image search and thumbnailing or something of the result page, you know, how would I do that? I could pre-compute the image thumbnails. I could, like, try to thumbnail them on the fly from the Larger images. What would that do? How much disk bandwidth that I need? How many disks would I do? Um, and you can sort of actually do thought experiments in, you know, 30 seconds or a minute with the sort of, uh, basic, uh, basic numbers at your fingertips. And then as you sort of build software using higher level libraries, you kind of want to develop the same intuitions for how long does it take to You know, look up something in this particular kind of hash table I use, or, you know, how long will it take me to sort a million numbers or something?

AI assessment note: “we had a data center in there. Um, so, I mean, I think this gets”