AI model

“Basically this one is still unsolved, which is, like, super easy. Actually, no machine can solve this one.”

“There's a lot of historical regulation around anti-discrimination and bias and things like that that, ah, certainly applies just as much to AI models as it does to humans and more simple analytical models.”

“Starting this year, we've now trained models, specifically an AI model, to understand the quality of a customer service conversation better than a human.”

“More transparency we can create also around, you know, how models have been trained, You know, the more research we have around the model, of course, the more trust it creates and easier it is for companies to adopt.”

“The way to think about today's models is they, they're like book smart, but not street smart. They're like a person who's read the whole internet, but they've never done anything in the world.”

“Because the cost will go down. So that's fine. Even if you're losing money today, you'll be winning some, I mean, earning some money tomorrow.”

“At some point, like we lost saturated on data that, that exists on the internet. That's interesting enough for models to keep improving on that. The next big capability jump is probably going to come from the models teaching themselves.”

“And so I'm generally in favor of finding ways to regulate end use cases and make the malicious use of the models be regulated and then punish that very strongly and make people responsible for the end outcomes that I am for banning the underlying technology.”

“The capabilities of these products are determined today by the capabilities of the models underneath, right? About 80, 90%.”

“When you're working in these domains that are, you know, like defense and medicine and finance, where it's like super high impact and you really have a vested interest in your models being right, you start to learn very quickly that a lot of the A lot of the u…”

“What we found is this sort of layout is particularly useful for debugging AI models because their errors tend to pop up in like localized regions of it.”

“And I think in within 10 years, the idea that you can be a sort of any science in any science without using AI models to help assist what you're doing will also be anathema.”

“First of all, the training of the people in your enterprise that are going to use this technology is as important as the training of the AI model.”

“And one of the points I make in the book Is I think we should pay a lot more attention to the interfaces around which, you know, we wrap these models. I think the utility of the models to some extent for enterprise applications will depend in a large way on, o…”

“I even think there's a future where these models can be a hundred billion parameters, but at, you know, have that intelligence of a hundred billion parameters, but then have the speed, latency, and cost of something that's still one billion or seven billion pa…”

“So I think we're gonna end up in a world where there are both Right? On both sides of the spectrum, some extremely large, powerful, generalist, costly models for some use cases, all the way down to a very specialized, simple, optimized models, and depending on…”

“We've seen a lot of companies who started their AI life cycle more with like a large model and then moving on to smaller models.”

“So the same way today, every company, every product has its own code base. Tomorrow every company, every use case is going to have its own model optimized.”

“Actually just with what we have, there's Plenty we can do. Plenty of great companies we can build. Just taking the current state of models and sort of deploying them in a way that's, you know, integrated in workflows and, you know, verticalized per industry.”

“So I do think that's a topic is getting increasingly important, especially as AI is being like first it is being used for more like high stack tasks, right? And a more complex task. And the second is, is like, it's now AI has increasing access to more tools an…”

“Just because a model, I think the second reason may be actually more important at least for now is that just because a model can fit in a million con token context doesn't mean that it can process that million token efficiently.”

“If you ask it to pull out data from a short document, it's going to basically be a hundred percent accurate on that. If you ask it to pull out data from a 200 page document, And you only give it one shot to do it. It's going to get 75% accuracy.”

“I think that the models are going to live next to the data in these warehouses, and we're already seeing that.”

“We don't think the intelligence get built on our platform. Like, I know that's a hard thing to say because everything's AI now. We believe that we're fundamentally mining the data and making it available so that the intelligence models, which are largely going…”

“But it still can't actually replace any of the software you use. It can't replace Excel, it can't.”

“Building a model gets more expensive, but the cost of using the model gets cheaper.”

“The thing that's become very clear, if it wasn't clear a year ago, is that the models themselves are sort of commodities in that, you know, there's half a dozen people who have a state-of-the-art model.”

“Is it better than the world's best doctor at prescribing drugs? Probably not. Will it get there? Probably.”

“I think we're underestimating how much you can do with the models that are out today, not the ones that are coming next year or next month, the ones that we have today where I think we're only touching the surface of what's possible with those leasing capabili…”

“And there's another emergent principle of you have to design from the model forward.”

“My advice to companies building is definitely build for what the model will be able to do six months from now, not for what the model can do today.”

“It's less about one single, like, algorithmic innovation that's gonna change the entire field and more about how do you make sure that, like, those pieces are set correctly because I think ultimately it's a complex puzzle that has many different parts and you …”

“That process can now be automated having a model that does it for you.”

“Over the next six months, over the next year, expect dramatic progress here. And like look at where we are now versus where we were a year ago. And the difference is I expect the same jump basically.”

“We'll probably reach like better than human on the GDP eval, and it won't change anything economically because It'll be all the connective tissue, and all the, like, you know, the context, and actually, like, the task won't be representative.”

“I think, like, in a way, I think it's getting more and more to be a thing of the past as the models are getting smarter and smarter. This is becoming less of a thing, but I think a few years back, and especially in GPT-IV days, this was the thing.”

“Just make your own internal benchmark that really represents what you care about, and then measure on that. And I think that's likely to be the most objective, most accurate way of measuring.”

“There's probably, yeah, not one individual blocker. And that's why we will continue to see sort of smooth incremental progress over model releases”

“I think my guess for that level of capability might be maybe 2027. I think we're probably not going to find out for quite some time afterwards because of the delay in getting prices. But I think by 20, 27, 20, 28, I think extremely likely that the models will …”

“The capabilities on cyber tasks of models are doubling every six months.”

“I think we have a hard time holding the point of view that models will reach the same level of intelligence and capabilities that the world's most capable people in every field have. And when you take a step back, and don't take the next 12 month view, but jus…”

“This thing, like how do models connect with the external world? It's a fundamentally very hard problem because, you know, when you connect in an unlimited way, you can break things in the real world.”

“The models are just, they're starting, like you see the first example they manage, so they've clearly made some progress, but they have not yet learned to do good reasoning in multimodal domains, and they have not yet learned to use one reasoning in context to…”

“So the understanding of what the models are doing on a higher level has progressed a lot, but then it's still an understanding of what smaller models do, not the biggest ones. But it's not so much that these patterns don't apply to bigger models. They do. It's…”

“But in the future, we are imagining we will have models that are More capable than humans, and even expert humans will not be able to reliably grade very complicated answers from the model.”

“In the mathematics I think we will see the models getting better on proving technical results, technical lemmas maybe including formalization and like things like lean the formal theory, improving language. I think the models, it's easy to imagine the models b…”

“It seems like as soon as we start kind of applying some optimization pressure, the models will learn to hide what they're doing from the chain of thought.”

“I am pretty confident we are not there at the moment. I think right now the models are still acting in isolated environments, and we are not seeing a lot of evidence for very coherent goals across different settings”

“You can see that the more capable the models are, the more likely they are to do this deception behavior.”

“In order to get those behaviors out of the models, you need to create somewhat of a contrived scenario or some special scenario. It's not necessarily something that we observe normally.”

“The models that we see today that we can play with today have been pre-trained on clusters that were built out a year or two ago. Because, you know, you need enough time to get the cluster running. You need enough time to do the large pre-training run. And the…”

“In a general purpose workload, crock. Grok doesn't work, right? You know, it can't train, it can't, you know, it can't inference really, really large models cost efficiently, right? You can't serve many, many, many users, but what it can do is it can go block,…”

“But if you've got a bunch of use cases where you need the right answer, as opposed to sort of the right answer, then saying that the model is better doesn't mean anything. I mean, literally, it is literally meaningless. What you're telling me is, I asked the m…”

“Not easy to pinpoint like specific things, but again, like, you know, this is just like my personal opinion and maybe I have colleagues and like the other people like sharing this with me, but I think we're underestimating how hard like jagged intelligence is …”

“In almost every lab the new generation of the models are built heavily using the previous generation of the models. I think that's basically like the case again, everywhere.”

“People don't experience average performance of these models. They experience, like, the failures. If you have your model doing a dumb mistake, the damage in the trust that it makes is, like, bigger than, like, you know, the benefit of getting hundred things ri…”

“Models, definitely, objectively, I would say, in a lot of scenarios we can measure, are safer than they were a year ago.”

“You can't just sort of trust models to get safer by getting bigger. You have to put in the work to actually make them safer. And this is, I think what a lot of AI companies are investing in. This is why we in fact do have models that are improving on these dim…”

“The problem is simulations are always going to be really hard and are not going to be truthful. So I think there will always need to be a certain, a little bit of training that will need to happen in the real world to make sure that the model realizes kind of …”

“Once you start having models that are really good, you accelerate yourself. Especially in terms of coding, given that we all code internally yeah, you accelerate yourself both for having these models, like train the other models, but also like build like the t…”

“But there's nothing, I would say, in the capacity of the model That is constraining the model to be as good at legal and like medical and like other domains.”

“And in general, I would say the performance of the model really depends on, like, the number of people who care about the final output of the model and who are looking at that model. So if they start looking more on specific verticals, like, these verticals wi…”

“There's part of the scientific process. I think that the models haven't been imbued with yet. And I'm sure people are thinking about how to do that. You know, like what, trying to get to what is the right question as opposed to here's a well-defined thing and …”

“For the next six months. Like, I think we'll see more of these sorts of math and science breakthroughs.”

“Right now they are far, far, far worse at engineering agent systems than they are at engineering most software. Far worse. because by definition, like that kind of work, which is so novel, has not seen a large amount in their training data. and so they have …”