The Wisdom Wall

“The other parts are architecture and data innovation. These also play a really, really important part in the Performance of pre-training and probably even more so than, than, than pure scale these days, but scaling is still, uh, an important, an important factor as well.”

“I think what might be happening instead is kind of a shift in paradigm where before we were kind of scaling in the data unlimited regime where, where data would scale as much as you would like. And we're kind of shifting more to a data limited regime, which actually changes a lot of the research and how we think about…”

“So at a fundamental level, you did, you do need the model to know about those things. So you have to train a bit at least on those so that it knows what those things are and knows to stay away from those, right?”

“In my experience, there's maybe one or two of those things that make a larger difference than other things, but it's really a combination of many, many changes and many, many things from, from a very large team that actually makes Gemini three so much better than the previous generations of Gemini.”

“We're not really building a model anymore. I think we're really building a system at this point. Um, people have sometimes this view that we're just training a neural network architecture and that's it. But it's, it's really the entire system around the network as well that, that we're building collectively.”

“I think over time that boundary has blurred quite a lot because we're working on these very large systems now. Research really looks like engineering and, and vice versa.”

“oftentimes we don't necessarily want to use the best performance version of a research idea, but we'd rather trade off some of the performance for a slightly lower complexity version because we think that will allow us to do more and more progress in the future.”

“especially in deep learning, a negative results doesn't mean something doesn't work. It means you haven't made it work yet often.”

“on the RL and RL scaling side, I think we're seeing a lot of the same things we're seeing in pre-training or we saw in pre-training. What's interesting here is because we have the experience of pre-training, a lot of the lessons apply, and we can reapply some of that knowledge to RL scaling as well.”

“the only way you really have to protect against cheating yourself and thinking you're doing better than you are is by actually creating held out evals and not really keeping them held out.”

“Bringing it back to the topics of pre-training, um, I think that the perception and vision side is very important for this, because now you're asking models to, to interact with computer screens. So, so being able to, to do screen understanding really, really well, Is, is critical. Um, and so, so that's, that's an…”

“I think this is, yeah, this is in general for, for vibe coding specifically, I think that's, that's maybe more of an RL scaling and post training thing where, where you can actually get quite a lot of data and train them all to do that really well.”

“First, I think a lot of progress has been made on this front since in the last few years. I think this is mostly around post-training, around search, use search tools and then make search calls, then they would have access to that new information.”

“So being able to understand how the stack works all the way down from TPUs to research is kind of a superpower, because then you're able to kind of find these gaps in between different layers that other people weren't necessarily able to see, but also to reason through the implication of your research idea All the way…”

“And then, so, so what that means is research in terms of how, how you use models and And the harness, et cetera, is becoming increasingly important and also how you make models and these harnesses more robust to making errors and recover from, from such errors.”