Foundation Model

“They actually are doing things that I think no other sort of big company would do. It's really, really well done. It's a really thoughtful UI. It's got a really powerful foundation model. I think it also is like, okay, wait, maybe this is not a two horse race …”

“That's just the first part of this, is meeting the robotics needs in the current technology, because we don't yet have a frontier foundation model that's robust enough for robotics.”

“And we're in pricing bake-offs with all the models every day. Like we pick the best model for the job and we're extracting every last bit of, you know, margin from them as we can.”

“Because we have foundation models for language, maybe vision, but not for physics. So, you know, the idea is instead of like right now what we've seen are narrow surrogates and we're trying to broaden their scope more and more, but ideally we have much broader…”

“These models are actually trained on this very broad data from the web, right? So these are human behavioral data. It's the, it's social media, Wikipedia, all these kind of data. So if you poke At the right angle, then you could see human behavior that would j…”

“My hunch here is I do think in the next Some number of years, we will start creating simulations that will actually cost as much as training a foundation model.”

“We're over-indexing on the model. I think models are great. But being able to take that model, package that, all that context, all that knowledge, all that training, and be able to deliver a solution to you is what the next big sort of revolution needs to be.”

“Percy was the person who literally coined the term foundation model.”

“Having having the need for a meaningful services component does, for example I mean, it serves a useful purpose also today, right? Like in a manner which might not have been valued as much a few years ago, which is that It absolutely gives you a certain moat a…”

“When you had models, you know, let's say a few years ago, you'd have to give it very, very tight guidance, very specific instructions that you didn't want it to deviate from, and as the models got smarter, you could kind of give it broader and broader guidance…”

“Most of the mode is not going to be technical unless you're a foundation model company or a world model. It's not technical.”

“The foundation models aren't built for that because they don't really understand persona. They don't have memory. You know, you think they don't have memory of multiple complex and the interdependencies between those. So I actually am seeing founders, some of …”

“The training run is not the expensive part. The training run is a very anticlimactic event, right?”

“What's happening for this field is that we are lacking a concrete definition of virtual cells and people almost equate foundation model With virtual cell, but in my view, virtual cell is probably a much broader concept than just foundation models.”

“I believe that foundation model or other more complicated AI models that trend on the right data will outperform these linear models in harder tasks, particularly in generalization tasks.”

“I do think over time, today a lot of the value has been accrued in the infrastructure and model layers. I do think over time we'll start seeing value getting accrued in the application layer.”

“And so like these foundation models and like how this works, it's not like you build it and then it just lives there. This thing has got to be living and constantly rebuilding itself.”

“The model alone, the foundation model alone, is, is not an AGI. No, you need the other guy, which uses all kinds of tricks to exploit The algorithmic information in the world model to come up with better plans.”

“There's a foundation model bets themselves, which are an order of magnitude actually two orders of magnitude, more capital intense than anything venture has done before.”

“Like anyone who's got a couple of billion dollars can make a foundation model which is what DeepSeq showed us as well.”

“Because what you're saying is keep up the price of AI as an input to the rest of the economy by protecting, protecting these foundation models pricing structure so that they can get a trillion dollars in revenue. And the loser, as always is the case with trade…”

“We would not build our own model. We don't want to compete with our customers, but we are building around the models so that enterprises can unlock the fullest value.”

“I think we're going to have that for science over the next two to three years. I do. Once discoveries start coming out and this field continues to mature even more.”

“The amount of money that you need to build a foundation model is two or three orders of magnitude more than anything else.”

“Every month, if you will, there is a new model, right? That's beating the old model, and visibly adopted already, and people are moving to the new models and the data is just flowing, like you're generating data, you're sort of getting access to new data creat…”

“I don't think foundation models. A product. I don't think a chatbot is a product. I think the value will be further up.”

“The models are kind of diff commodities, and the chatbot isn't the right UI or the right product, and the companies aren't going to be able to build all of that stuff themselves, so therefore they're low-level infrastructure.”

“Like, so in other words, another asset test is, you have an eval that's private. You're using a model A. Can you switch it to model B and, you know, climb up? If you can, then you're in control. If you can't, you're not in control.”

“And so it does sort of seem to me that like, if the chatbot isn't the UX and it needs to be apps and the model companies aren't going to build that. And the models themselves are basically commodities, as, at least as you can see them as users, then why would …”

“Interoperability model routing based on request being model provider agnostic on the foundation model side is where I see this going. It's not gonna be one, and I mean, we've talked about this for a long time, one model to rule them all, systems that can route”

“If you are fighting against them, you've already lost. If you haven't figured out how you're going to win with them, how you're gonna not just coexist, but actually find ways to collaborate potentially, and if the tailwinds that they create are not yours to le…”

“These intelligent systems, these AI systems are not just a model, right? There's a lot of layering that happens on top of these models. These systems have to really deliver specific capabilities or specific experiences that help people do certain specific task…”

“People keep saying like the cost of switching between models is zero. It's cheap, but it's not zero because sometimes you like spend, you know, three, four months, like fine tuning exactly how the model should be, or like how the, Product should be. And then o…”

“To the extent that that is encoded in a model, I think a lot of their business will be at risk, but to the extent that it is encoded in workflows that is where they will be able to develop mode.”

“And I think a lot of those ideas, both from AlphaGo and AlphaZero are really, really relevant to where we are with today's foundation models. And I think a lot of that is what we're going to see of the advances the next few years.”

“That all the leading foundation models are pretty poor at games, which is quite interesting.”

“I think that those kinds of interdisciplinary teams, especially if it involves the world of atoms as well there's not going to be a shortcut to that, at least in the foreseeable future. Those are areas that are pretty safe from just getting swarmed by whatever…”

“This only works for simple cases today, and the reason why that's the case is because I think it's pretty fundamental limitation of the model that we have today, which is that they are not at the core model that take action in the world and see the consequence…”

“I don't think it's going to get replaced. I think it's going to get built on top of these foundation models, just like the way we do with our world models.”

“And I think something like that might happen in the world of robotics, but it can't happen today because if you want to put together some cool new robotics applications, some cool new robotics idea, you kind of have to build this, like, monstrous stack, and yo…”

“There are the approaches that work well for robotic manipulation that often are the opposite. They often use very little simulated data, often use large amounts of real world data, and very large foundation models.”

“It's only in my hands the one I'm still not naming is only about five times faster than my fastest DFT calculation on a GPU, and it also doesn't work all the time.”

“And the problem is that as far as we can see, there is no winner takes all effect or network effect in a foundation model.”

“I don't think it's actually the right metric, like the single right metric to focus on anymore, because there's a lot of examples with these LLM wrapper companies where it's like temporarily you can appear to be magic for a lot of the market who either doesn't…”

“Cause right now I think like the models are already so good that the constraining factor is actually your ability to provide them the context information to win.”

“It might be that like the next stuff that like is soda on benchmarks is not the most expensive newest foundation model with the most like GPU training. It's like a swarm of lower cost cheaper models working together just like humans do to solve a problem.”

“The underlying tokens Probably can come from anywhere and how much margin there is in that business over time probably does look a lot like power, like Sam's talking about. And the real value is going to be in the orchestration.”

“For example, at future house, we took the opinion that scientific agents are the future, and that allows us to skip a lot of steps because a lot of other people were like, we need to build a foundation model for X. Yeah. And we just skipped all that.”

“And then from an AI standpoint, you know, I feel like we're generally okay with the investment. I mean, there's so much money going in there that that's like a well-funded area, both the foundation models and the derivatives that are trying to do specific apps…”

“Right now, currently, I would say that the complexity of the application itself is very high and is not subsumed in the foundation model. And so I think these things are not as straightforward as they appear and the benchmarks can be misleading.”

“Lightspeed From a very early on in, in sort of the AI kind of revolution took this position that actually AI and foundation models are more akin to software and code than they are to business lines and strategies.”

“If you're a huge foundation model and you can kind of hit every layer of the stack, there's probably more revenue for you to take advantage of than if you're just at the application layer.”

“I think if one thing happened this year, I think more and more folks coming over to team products that models aren't going to solve everything.”

“I think you know, a third area is the next set of foundation models are going to come, and by that I don't mean the Neolabs and the Next Gen LLM, which of course will happen, but I mean, physics, materials, science progress by models, math progress. And I thin…”

“I believe that even if we had no more progress today with models, which is absolutely not the case, but if, even if we had no more progress, we are way behind on product. There's so much more product to build.”

“Pre-gen AI, the biggest thing was as a seed stage investor, is this real value creation? Like, is this a nice to have versus a must have? That was the big question for most companies that came to pitch... These days, the reality is AI has put founders in a pla…”

“There's no, there's going to be no technical differentiation in these foundation models, right? Unless something crazy happens, but like, you're just gonna have a bunch of models that are quite good, in which case you're competing for attention.”

“The future of a company is that company has its own foundation model that captures essentially the tacit knowledge that makes the transactional costs of how knowledge gets accrued and diffused inside the organization faster.”

“I worry perhaps for startups that a lot of the things that they offer that are these sort of like niche carve out experiences for a category will be replaced by the broad based foundation models. And that many of these companies that I've seen a very rapid ris…”

“Incumbents have the advantage of a lot of data, right? But they also have the baggage that often the data is not very clean. So as you build your models, as you build your foundation models, et cetera, are you relying on good data or not, right? Whereas a star…”