model

“As models get bigger, they fine tune faster. So they're more sample efficient as they get bigger.”

“At the time, the models hadn't been trained at all to be faithful to a text. So they were just generating. So then when you ask them a question, they tried too hard to ask, answer the question, and didn't try hard enough to answer the question given the text o…”

“It is very straightforward to take a model and say, parse through all of these documents and find second order derivative bets and say, oh, it turns out that energy is like very, very adjacent to investments in AI and may not be priced in the same way that GPU…”

“It can turn the compiled source code, which is impossible for any human to understand into the Python code that is entirely reasonable to understand. And, you know, it doesn't run. It has a bunch of problems, but like, it's so much nicer that it's immediately …”

“If you make assumptions about the capabilities of models, and you engineer around them, you're almost, like, guaranteed to be screwed.”

“It feels very clear to me that this type of logic is going to be built into the model. Anytime there is control flow complexity or uncertainty complexity, I think the history of AI has been to push more and more into the model.”

“And the reality is that if you look at Notion, Airby does the job of taking Notion and putting it in a structured way, but that's a way that is not really usable to actually make it available to models in a useful way. Because you get all the blocks, details, …”

“And you can see here for every node, you can configure which model you want to power the node.”

“Even if your model is theoretically more efficient, if somebody goes and runs it and it's two times slower one of the things that, that we've learned is that if you're in that situation, it's just going to be dead on arrival. So you want to be designing your a…”

“The first way is this, the simplest way and the newest way, but it doesn't work very well, which is you take a screenshot of the website and then you click on a particular pixel value on the website and like models are not very good at that at the moment. Like…”

“In my opinion, like, people in, like, few years will click On, like, websites way less. I want to see the plot of, like, website clicks over time, but then my prediction is, like, it will go down and, like, people's access to the internet will be through the m…”

“Pokemon's actually really nice because, like, if you don't do anything for five seconds, like, there's typically not a consequence by the nature of, like, doing inference on a model every, like, snapshot of time. It's actually a pretty good game to be able to …”

“Like, we've seen this pattern in Cloud 3.7, which is kind of different from the other models where, like, it really loves to gather up context first, and then it starts doing edits.”

“Everything is the model. Like that's the thing that wins in the end. And it just, as the model gets better, it's it subsumes everything else. So, you know, at some point the model will encode its own knowledge graph. It'll encode its own like KV story if you j…”

“And that, if you can crack that scaling direction of, like, pushing the boundary of how long these models can go out and do these things for, that is a plausible path towards things that become marvelous.”

“But our take in general is that freezing the model at a specific quality level and freezing the model at a specific data set just feels like it's lower leverage than continuing to iterate on all these external systems.”

“It's actually like only a few models are able to speak two languages in the same sentence and then pronounce them properly.”

“As the models get better, people are going to find themselves using natural language, working with an agent more and more and less being in the weeds and editing code and tab autocomplete.”

“I think part of the intuition of RLVR is that the model is good at knowing which prompt area it is, which is why the models don't get worse on knowledge benchmarks if you're trading on like just math or precise instruction following. So the model just kind of …”

“In order to get them to be able to operate according to different preferences and values, we just need to show that to them during training, and they are able to sort of generalize to different preferences and values that we didn't actually train against, and …”

“We're wasting a ton of capacity in these models on knowledge that is just totally unnecessary for them to have.”

“Most of the models that the vast majority of people will be using in say three years will be single digit B or smaller.”

“The models don't see as well as they could. They see, okay, you know, you ask them analyze a complex photo and they're able to do it, but I want them to be as persnickety as a like really good visual designer. Like, no, that looks, the baseline looks a little …”

“Initially, you know, our models are trained the way they were trained to use tools, and that kind of bakes in a habit, and so we've been getting the models better at using different types of tools.”

“MCP is a protocol between the AI application and like servers, right? So the model is actually technically not involved in MCP.”

“It seems like we're kind of, like, a few orders of magnitude of, like, kind of data efficiency, basically, away from, like, that kind of, like, you know, you do something once, or, like, you make a mistake like, you, yeah, you introduce, like, a bug in your co…”

“I think it only applies to models that were initialized from the same starting Z. Usually, yes.”

“Will those gaps still exist in the next few generations of models? It's like a little unclear to me though. Because right now these like hyper optimizations we make, I'm not sure for how long they're still really relevant.”

“I think we're going to see a lot of like applications and companies that do very impressive things with AI that in the short term might seem very effective because they're very specialized to individual use cases. But I think once models get better at generali…”

“And I think the issue is that many such problems now, I would say these models can probably crush. Yeah. These are problems that we usually take again, you know, timescale for a theoretical physics paper is six months to a year. That's pretty typical.”

“From our experience, the heuristic engineering also have the models get absorbed into the models themselves.”

“And it turns out the models are absolutely horrible at this. No one scores statistically better than random chance.”

“If you make, if you're testing the model for robustness or safety, right? And it's aware that it's being tested because you've set things up in a very artificial way, right? Like the email addresses are at example.com. The webpage is clearly not a real webpage…”

“There were a couple of models that seemed to be very, very robust, right? Like the red teamers found just a handful of successful breaks on them.”

“When we look at our kind of three-year roadmap, right the end goal that we want to reach is one where You know, the models are just doing end-to-end research, and I think a part of that problem is just being able to have the model come up with good taste.”

“And so again, the core bet that we're making is that is true for science. That if the model is trained on an increasingly broad set of data, the amount of data that you need in a given domain, that data requirement is reduced. In some cases will be reduced to …”