Google DeepMind

“Google DeepMind has also applied this approach Do things like code optimization, where essentially they're searching for more efficient implementations of a particular algorithm. So they shaved about seven percent off the sort algorithm in the C++ library, whi…”

“He was very enthusiastic about agency and sort of, and I think in part because agency When you start thinking about training AI agents, it gets back into potentially a zone for reinforcement learning, which was what DeepMind was always known for.”

“But if you look, Gemini is like, you know, a thousand people, there's a, there's still like 10,000 plus people doing all kinds of really like longterm foundational research at DMI.”

“And I think if we had done it a few months earlier, we would have probably lost.”

“Oftentimes we don't necessarily want to use the best performance version of a research idea, but we'd rather trade off some of the performance for a slightly lower complexity version because we think that will allow us to do more and more progress in the futur…”

“At a super high level so we have a pre-training team, a post-training team. On the pre-training side, we have people working on the model, on the data, the infrastructure, evals as well, very important.”

“The first part, I think, especially in the next year with more agentic workflows being enabled more and more, that should be able to really accelerate our work there.”

“I think deep down, I do believe that the long-term answer is to learn this differentiable end-to-end way, which means probably doing pre-training or whatever that looks like in the future, Learn to retrieve as part of the training and learn how to do search as…”

“And I think that's a mindset that has really evolved over the last few years at DeepMind, especially where maybe there was a bit more of the traditional research mindset before, and now with Gemini, it's really more about research engineering.”

“For us, at least on the attention side, we've made some really interesting discoveries recently that I think will shape a lot of the research we do in the next few months, and I'm personally very excited about that.”

“Rather than just doing compute in, in the depths or in, in the model side, you also do compute and allow the model to think more on, on the sequence length side of things. So the model actually starts to form hypotheses, test hypotheses, invoke some tools to v…”

“It's still remarkable how much progress we're able to achieve in this way, and it's not really slowing down.”

“There's actually very little of that. I think because all of the leadership has a research background that they're very much aware that yes, to some extent you can force and accelerate specific benchmarks and certain goals, but in the end, the progress and the…”

“At the high level, I don't think the architecture has changed that much compared to the previous one. It's more of what I was saying before, where a few different things come together to gather, give a large, large improvement.”

“And then, so, so what that means is research in terms of how, how you use models and the harness, et cetera, is becoming increasingly important and also how you make models and these harnesses more robust to making errors and recover from such errors.”

“So it's a fairly large team at this point. It's a bit hard to quantify exactly, but maybe a 152 hundred people I work on a day-to-day on the pre-training side between data, model, infrastructure, evals, and so coordinating the work of all of these people into …”

“The other parts are architecture and data innovation. These also play a really, really important part in the Performance of pre-training and probably even more so than pure scale these days, but scaling is still an important factor as well.”

“Bringing it back to the topics of pre-training I think that the perception and vision side is very important for this, because now you're asking models to interact with computer screens. So, so being able to do screen understanding really, really well, Is, is …”

“In my experience, there's maybe one or two of those things that make a larger difference than other things, but it's really a combination of many, many changes and many, many things from a very large team that actually makes Gemini three so much better than th…”

“I think, if I'm being honest with myself, I think we're ahead of where I thought we could go.”

“So at a fundamental level, you did, you do need the model to know about those things. So you have to train a bit at least on those so that it knows what those things are and knows to stay away from those, right?”

“Not easy to pinpoint like specific things, but again, like, you know, this is just like my personal opinion and maybe I have colleagues and like the other people like sharing this with me, but I think we're underestimating how hard like jagged intelligence is …”

“Like in a safe setup, you know, where they can like, because right now we definitely, we don't, we're not confident about, you know, them doing the right things all the time and measuring like how much they can push and how long they can push a task is very di…”

“I think that the AI company CEOs understand that they're on the path of building wildly smarter than human systems, like super intelligent AI systems. They understand that we don't really have a like clear thought through plan for how to manage the risks from …”