foundation model

“And models today, they're not optimized for reasoning. It turns out that there's not actually that much explicit reasoning data on the internet.”

“So I think internally, yeah, we have a lot of thoughts on what reasoning is, and we generate a lot more specific data. We're not just like, oh, it'll figure out reasoning from this black box or like, it'll figure out reasoning from the data that, that exists.”

“Code is the most explicit example of reasoning data on the internet. Yeah. And it's not only structured, it's actually very explicit, which is nice. You know, it says this variable means this and then it uses this variable and then the function does this. Like…”

“So I think that we continue to feel like graphics and these foundation models for anything really related to pixels, but also definitely images continues to be very under invested. It feels a little like graphics is in like this GPT two moment, right? Like eve…”

“There aren't really big data moats around foundation models. They are built from a subset of the web. What is difficult is the cost of capital to build these, and what, one of the ways in which you can reduce this cost is by making more efficient systems.”

“In a world where foundation models are looking more and more commodity. And if, and I think a huge amount of gain is going to happen from how do you use foundation models as like the, like well learned behavioral cloner to go solve agents.”

“The way we are building Elicit is not let's train a foundation model to do more stuff. It's like let's build a scaffolding such that we can deploy powerful models to good ends.”

“The incentives, the economic incentives for companies to train their own foundation models, I think, are diminishing. So the, like, window in which you are the dominant pre-train, and let's say that you spend five to forty million dollars, you know, for like a…”

“We're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models.”

“People are building too many vanity foundation models rather than taking better advantage of fine-tuning”

“The model is like this sort of like 10 X multiplier. You're, how good the bottom line model is, huge, huge swing. And then kind of what you can do on top of that, you can squeeze out three, four X kind of more. And so that's kind of where the realm of, you kno…”

“I think the idea that a company can make money sort of pre-training a foundation model is probably not true. It's hard to, you're competing with just you know, unreasonably large capex budgets.”

“After that, I think, you know, probably within six months to a year, we're going to start to see the really big foundation models being trained in this way.”

“So, so it's very similar in terms of how you interpret results. You want to treat everything as a black box. You want to establish your bounds, you know, what are you, and I've had tons of experience in algotrading, both from making money and not making money …”

“And if you look at the difference between the same model and the biggest spread across agent frameworks versus the same agent framework and the biggest spread across models, It's much larger across models, which I think goes to show that like improving models …”

“Then you have a foundation model. Each, you know, charge becomes this like dense embedding. You start to see these Clusters sort of pop out and, you know, in real time that they're card testing and you can block them. So yes, it is happening on the charge path…”

“Now you can think about, okay, actually foundation model, text alignment, like human readable description of like why we're worried about this charge. And then today, a human tomorrow, an agent sitting on top of that and decisioning, like reasoning over The mo…”

“For example, at future house, we took the opinion that scientific agents are the future, and that allows us to skip a lot of steps because a lot of other people were like, we need to build a foundation model for X. Yeah. And we just skipped all that.”

“It's only in my hands the one I'm still not naming is only about five times faster than my fastest DFT calculation on a GPU, and it also doesn't work all the time.”

“People keep saying like the cost of switching between models is zero. It's cheap, but it's not zero because sometimes you like spend, you know, three, four months, like fine tuning exactly how the model should be, or like how the, Product should be. And then o…”

“Like, so in other words, another asset test is, you have an eval that's private. You're using a model A. Can you switch it to model B and, you know, climb up? If you can, then you're in control. If you can't, you're not in control.”

“I think we're going to have that for science over the next two to three years. I do. Once discoveries start coming out and this field continues to mature even more.”

“I believe that foundation model or other more complicated AI models that trend on the right data will outperform these linear models in harder tasks, particularly in generalization tasks.”

“What's happening for this field is that we are lacking a concrete definition of virtual cells and people almost equate foundation model With virtual cell, but in my view, virtual cell is probably a much broader concept than just foundation models.”

“The training run is not the expensive part. The training run is a very anticlimactic event, right?”

“My hunch here is I do think in the next Some number of years, we will start creating simulations that will actually cost as much as training a foundation model.”

“These models are actually trained on this very broad data from the web, right? So these are human behavioral data. It's the, it's social media, Wikipedia, all these kind of data. So if you poke At the right angle, then you could see human behavior that would j…”

“Because we have foundation models for language, maybe vision, but not for physics. So, you know, the idea is instead of like right now what we've seen are narrow surrogates and we're trying to broaden their scope more and more, but ideally we have much broader…”