open models

“Most RLHF is only a few epochs, at least in the open models”

“I would say right now the top five models on our inference stack are probably all fine-tuned versions of open models.”

“The benefit you can get out of that is you could build a, Better open model, often smaller, often easier to do inference if you know what you want, right? So I think the whole trade-off would be, and the x-axis would be how generic the hosting will be. The y-a…”

“And I think the problem is this blocking or ideas really, it impacts people in different ways. It disproportionately helps companies that have a head start, which are usually the closed labs, and it hurts incoming newcomer players where you either have now to …”

“You know, if you remember the beginning of this year, it was all about bio-risk of these open models. The whole thing fizzled out because there's been, finally there's been, like, rigorous research, not just this paper from coherent folks, but there's been rig…”

“I think there's a lot of investments in companies that at the moment are releasing their model in the open, which is really cool. But it's usually more because of commercial interest and not wanting to support this like open models in the longterm.”

“There is a large swath of research on modeling, on how these models behave, on evaluation, on inference, on mechanistic interpretability that could not happen at all. If you didn't have open models.”

“But for open models, yeah, you can do better because you have access to logits, so why not use them? At least that's my take.”

“If open models are to win, part of it could be just, like, everybody can have exactly the model they want. We're serving GPT-IV. It's kind of its thing. You can prompt it, but if fine-tuning is more effective than prompting, everybody can have the model. That …”

“I expected. It'll be best in class for some size Category in some subset of tasks. That's like, OpenAI only does things like that.”

“Like the local modeling community, I think is much smaller than people give it credit for, because most of the use for open models is still in APIs.”

“The other thing I always think about from a finance point of view that, Maybe people don't even think, don't think about that closely, but obviously push back if you disagree, is that it's basically a proxy for the gap between open models and closed models. Be…”

“I feel like this is definitely a form of lock-in because you ideally want to be able to run prompts across multiple providers and all that. And yeah, caching is a hard problem. Like, I think ultimately, like, you control your destiny if you can run your own op…”

“Yeah, I mean, I think we're state of the art for open models.”

“So I feel like this always ends up being a tool call, a hardness issue. Then, you know, an actual model issue.”

“Claude is actually really, really lenient for tool calls. So even if, you know, your coding agent harness messes up, it can figure out that, oh, I'm being sent this error and can fix itself. Not the case with you know open models”

“We are not at a level of capability right now. That we should start restricting, you know, open models in any way, shape or form. I think it will hurt innovation if we do so.”