Closed Source Models

“When you take a closed source model, it does really well on a benchmark. When you wrap it in its own harness, it decays in capability. When you take an open source model, And you use any other open source model, it improves in capability.”

“All of that is, like, maybe not all, let's say, 90%, to be fair, is made by closed source model, right?”

“This is made doubly worse by some of the best open source models were made by Chinese companies. And they are exceptionally good models. Kimi Ketu, Deep Seek. When the GLM, these are extraordinarily good models. They're not quite as good as the closed source m…”

“When customers use this off-the-shelf closed model, what's very sad is that they are not leveraging, you know, these data that they have been collecting for four years, or sometimes for decades. So much data, sometimes it's trillions of tokens or data in a ver…”

“On here we can sometimes build something 10 X cheaper by just fine tuning a model, and it would be better on prem, on their own server, and also much cheaper as well”

“There will be areas, for example, like software development, right, where there is so much horizontal aspect to it in terms of how it is practiced across the world. Where I do think these other approaches, like, you know, where what you bring to that is the co…”

“I think for sort of AI companies like Cognition, like all your competitors, like AlphaSense, like dozens of others, they are trying to replace closed source models with very, very fast open source models, and they're trying to drive the open source Accuracy dr…”

“Like I've never seen a closed source model that can create a good LoRa ecosystem. It just basically doesn't exist.”

“It's a very traditional form of regulatory capture. If you have a big pharma company, they work with the government to prevent other entrants into the industry. And this is exactly what it felt like is at least partially happening. In the AI world. Now that ma…”

“Most businesses don't love using closed source models. They want to use open source models. For all kinds of reasons, you know, privacy, security, continuity, cost”

“You can evaluate a model much better if you have access to the weights than if you only have access to APIs. And so if you want to build certainty around The fact that your system is going to be a hundred percent accurate, I don't think you should be using a c…”

“And that's our prediction is With specialization, there will be a lot of expert models, really, really good, and even better than, like, one size fits all open source closed source model.”

“And this might be a little controversial, but like, I find a lot of arguments, ah, based on whether, like, closed source models are safer, or open source models are safer, very much related to whether, what kind of cultural, ah, culture they grew up in, what k…”

“Clem, CEO at Hugging Face, a few days ago, forecasted that in 2024, open source models are going to reach the level of the best closed source models. That may be aggressive for 20, 24, but they are headed towards parity.”

“That needs the open source models or fine tuned versions of them to match the closed source, the closed ones. And That's not the case today.”

“Even if there exists a checkpoint today, that's as good as the close models. It's still not an incentive to use them and serve them yourself because you're going to burn more money than paying for the close ones because the close ones are being served in such …”

“So, there's lots of stuff around steering language models at the embedding layer itself, and not using text but this is not yet exposed To at least closed source models, so.”