Open Models

“People hear, you know, open model and immediately want to train their own model, and I get it. I get why. I was actually the same way. It sounds really cool, but I feel like that's like an advanced move. The practical move, the beginner move, where you should …”

“Well, the easy answer is fifty-fifty in the same way that what percentage of enterprise software today is open source versus proprietary. I think it's a pretty even distribution.”

“There is going to be probably a difference between the commodity task executors. So this is just like your everything model and that we think will be dominated by open models.”

“I think that probably the consumer will continue to follow the sort of like most cost effective solution. And so what does that look like in one, two, three years? I think it's open models. I think the most cost effective solution for a model is going to be th…”

“I think close to, I think almost all usage of models in three years are going to be primarily open, but that difference between frontier and open is going to become actually larger than it is today.”

“Their whole pricing power, that premium they're able to charge is because They're six months ahead of those open models, and that six months would be eaten up in the blink of an eye if they were subject to regulatory approval.”

“I'm not convinced that 90% of enterprise workflows can be handled by open models or Frontier models right now.”

“We are not at a level of capability right now. That we should start restricting, you know, open models in any way, shape or form. I think it will hurt innovation if we do so.”

“So within, ah, within Firewalls, obviously, we dock for our own product. We use OpenModel to drive our recruiting process, candidate sourcing, and the feedback collection. We use OpenModel to even drive some internal finance processes. Obviously, for coding, w…”

“My hottest take is that in the next 12 months, 90% of tokens will be going to open models.”

“At the beginning of the year, we saw enterprises were using less than one percent open models, like less than one percent of their tokens were going to open models. Then in around March, it kind of crossed that one percent threshold. And then in May, it crosse…”

“So like Decagon, the customer support AI company, they said that 90% of their usage now is being sent to Open models, but those are open models that they've had the opportunity to post train on and do a huge amount of customization based on all of their learni…”

“So, and the, this growing collection of open models includes some of the best models that are already on par or surpass, let's say, Plot Sonnet or even GPT-A for Mini.”

“With the right harness and the right amount of test time compute, you can use open models to find all of the vulnerabilities that Fable can find.”

“Absolutely. Because it's in your incentives to bring down the cost.”

“A lot of enterprises will realize so many of the tasks that we're doing, we don't need the very frontier to do it. Like, and we can do it much faster, much cheaper with these open models.”

“There's kind of an ego thing, where, oh no, no, the work that I'm doing, only a frontier model could handle. Oh, this mere open model can't deal with the work that I'm dealing with. And this is like, even admittedly, when I first started switching over, I'd be…”

“So I feel like this always ends up being a tool call, a hardness issue. Then, you know, an actual model issue.”

“Claude is actually really, really lenient for tool calls. So even if, you know, your coding agent harness messes up, it can figure out that, oh, I'm being sent this error and can fix itself. Not the case with you know open models”

“Yeah, I mean, I think we're state of the art for open models.”

“Nvidia sells GPUs. They don't sell proprietary models. And if they have great open models that everybody can use, then everybody is going to buy GPUs.”

“If you have open models, you can actually add new parameters. You can make a lot of deeper things that you cannot do with closed models.”

“I feel like this is definitely a form of lock-in because you ideally want to be able to run prompts across multiple providers and all that. And yeah, caching is a hard problem. Like, I think ultimately, like, you control your destiny if you can run your own op…”

“I think the best ones will always be closed but if you want maximal flexibility and some kind of open ecosystem around it might be that open ones are the ones that most people choose.”

“The other thing I always think about from a finance point of view that, Maybe people don't even think, don't think about that closely, but obviously push back if you disagree, is that it's basically a proxy for the gap between open models and closed models. Be…”

“If open models are to win, part of it could be just, like, everybody can have exactly the model they want. We're serving GPT-IV. It's kind of its thing. You can prompt it, but if fine-tuning is more effective than prompting, everybody can have the model. That …”

“Like the local modeling community, I think is much smaller than people give it credit for, because most of the use for open models is still in APIs.”

“I expected. It'll be best in class for some size Category in some subset of tasks. That's like, OpenAI only does things like that.”

“The Chinese AI labs are the world, world's leading open, open model companies. They offer the most advanced open models.”

“If you look at both open and closed models over time, you know, the ELOs are just up and to the right. The open ones have converged and even in some cases surpassed closed models.”

“But for open models, yeah, you can do better because you have access to logits, so why not use them? At least that's my take.”

“And it'll actually be an advantage for like the agent builders to not be tied to a specific model and be able to like, just jump around between whichever model is best. And I think once that start is happening, we're going to have like that, that's going to me…”

“There is a large swath of research on modeling, on how these models behave, on evaluation, on inference, on mechanistic interpretability that could not happen at all. If you didn't have open models.”

“And I think the problem is this blocking or ideas really, it impacts people in different ways. It disproportionately helps companies that have a head start, which are usually the closed labs, and it hurts incoming newcomer players where you either have now to …”

“You know, if you remember the beginning of this year, it was all about bio-risk of these open models. The whole thing fizzled out because there's been, finally there's been, like, rigorous research, not just this paper from coherent folks, but there's been rig…”

“I think there's a lot of investments in companies that at the moment are releasing their model in the open, which is really cool. But it's usually more because of commercial interest and not wanting to support this like open models in the longterm.”

“And I do think that open models will immediately have their guardrails breached.”

“I can definitely confirm that the data clouds are paying a revenue share. To the open models.”

“The benefit you can get out of that is you could build a, Better open model, often smaller, often easier to do inference if you know what you want, right? So I think the whole trade-off would be, and the x-axis would be how generic the hosting will be. The y-a…”

“I would say right now the top five models on our inference stack are probably all fine-tuned versions of open models.”

“Most RLHF is only a few epochs, at least in the open models”

“As part of that, every nation will need their own data sets, which again, have from broadcaster data. They will need their own open models that can stimulate innovation internally as well.”

“As part of that, every nation will need their own data sets, which again have from broadcaster data. They will need their own open models that can stimulate innovation internally as well.”