Frontier Models

“We've built the leading frontier model for epigenetic reprogramming.”

“If you miss the frontier models, back to your first question as an LP, but you kind of were along the benchmark, maybe slightly below the benchmark, you're keeping your job.”

“In an ideal world, what you want to do is actually take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow.”

“The answer there for every startup that I know outside of a very few. Is, yeah, it's, well, we're hoping the price, we're hoping the cost curve makes it efficient in 18 to 24 months, right? In the meantime, you're subsidizing in part, right? Because that's wha…”

“I think what we're going to see is a split between job functions that demand kind of mid IQ intelligence, and those will often be open weight, sort of biased with reinforcement learning, you know, things that make the models even cheaper, more performant for a…”

“Well, the easy answer is fifty-fifty in the same way that what percentage of enterprise software today is open source versus proprietary. I think it's a pretty even distribution.”

“I think that the TAM of frontier models is frankly, overweighted right now.”

“I think close to, I think almost all usage of models in three years are going to be primarily open, but that difference between frontier and open is going to become actually larger than it is today.”

“I mean, there's only going to be one or two trillion dollar outcomes per decade, and they're likely going to be all concentrated for now in, you know, frontier models.”

“If you look at a modern MOE model, there are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated, and I think the frontier models are closer like one percent.”

“What's happening is, is that you can't customize anthropic models or open AI models right now. Not the big frontier models. You're not allowed to do that.”

“And if you're looking at a Frontier model with double dollar digit cost per token, and you're looking at an open source model that's 10, 11 cents per token, while all tokens aren't created equal, it's still enough of a difference that there's going to be a mas…”

“They're going to replace frontier models. I don't think you can bolt on continuous learning into an existing frontier model. I think they're going to try and the early stages will look like that. But I think ultimately it'll call for a new form of architecture…”

“Where in some cases, the model performance of frontier models go all the way down to 20, 30%. Especially if you go into that more niche population on topics that our customers will actually care about. On more gen pop, it might be around 50 to 60%.”

“I think they're more nervous about frontier models, usually. Part because there's just like a much, there's much more confusion around the data policy about what's like actually happening to the props that I'm sending and where they're being stored and how the…”

“This is what open weight models are generally really good at compared to frontier models. When you have a deterministic task where you know the shape of the output, you know the type of problem that you're working on, and it's a type of problem that has been s…”

“Because of the guardrails and because of the question around biohacking and using this model to generate like the access right now for people just to take it is very, very limited once you want to ask some biology question. And so basically most of the life sc…”

“There was a research published recently about what they're calling kind of like universal type of squats or universal hallucinations where all the frontier models all have the same make the same mistake and sort of assume there are certain packages that exist …”

“The frontier models are gonna cause, you know, a, you know, they are causing kind of a massive reduction in the time between the vulnerability discovery and vulnerability exploitation.”

“Customer support will not be using frontier models.”

“And if you fine tune it to be really good at that task, it can be just as good or better than the big models.”

“So we think for jobs like that, Frontier models that are very smart, that can try out a lot of things, make a lot of sense.”

“If we ran our entire business exclusively on frontier models, I would care a lot about costs, and I would think a lot about that, but once you're already, once you've already made the jump to saying, okay, now, now we know how to think about open source models…”

“The whole idea of, like, if the, if an amazing intelligence, an amazing model is made, should it be widely available? That's actually at odds with distillation, because you have to hunt those bad actors down, and you play this whack-a-mole.”

“Though for what it's worth, on whatever chips they have now, they've built frontier models.”

“And even despite, you know, Kimi K three being said is a token hungry model. It's not as cheap as other Chinese models. The task per, I think the task per token usage is still quite high. It's still cheaper. It's still cheaper than the most frontier models and…”

“For an, a collection of agents to do all of that in kind of a one shot push button go in an environment that's changing and actively fighting back. We're nowhere near that. Right. At least on the frontier model side.”

“One thing we say a lot on the team is you need frontier products in order to have frontier models and for people to feel the magic. Of frontier models.”

“I'm not convinced that 90% of enterprise workflows can be handled by open models or Frontier models right now.”

“I'm not only taking the frontier models and just making them available. I'm doing it across classification levels. I'm combining some of the most, you know, important data that the United States has from an intelligence standpoint, we're deploying it to warfig…”

“The frontier models are doing an exceptional job already in helping test the security of networks. And they're changing so fast, Joe, that I'm very confident regardless of which frontier model you use you're gonna find it's outpacing humans doing the work, you…”

“They're not state of the art compared to the frontier models, but they're, you know, six, nine months behind, depending on how you measure it.”

“Is our frontier models required for the majority of tasks that are going to be done and become agentified? And I don't think they are.”

“I think this is actually going to be a moment where even more so the idea that Frontier models are a massive moat really goes away.”

“Jesse's explanation, I think, is the correct one, which is that open source is good for production grade, like production scale workflows. Yet we're so early in this AI journey that there's not that many production grade workflows. And so if you are trying to …”

“What Demis, the AI leader, Google, Sabis, proposed was, like, It was this idea that there should be a separate government body that reviews frontier models basically for 30 days prior to their release.”

“But two is like, for the agentic tasks of 18 months from now, inside those major repositories of knowledge, and asking the models more and more things in underspecified ways, I suspect that the accuracy of the models would go down. The phenomenon of context fr…”

“And now you can solve these tasks with frontier models and compaction. And when you ask them to do so, they will consume thousands of dollars for queries that we think are harmless. That every employee in the company would be able to answer.”

“What we've observed is about six months after the latest and greatest model comes out, you'll have an open weight model that is just as good and maybe costs, right, one 100th to go and use it”

“People are speculating that the intelligence gap between that commodity stuff and the frontier stuff is going to collapse to the point that people won't pay for the frontier stuff. There is no evidence of that on the field today. It may develop over the course…”

“The open router, you know, all the tokens with L, you know, with open source is a little misleading, because all the tokens can be in one place, but all the dollars can be in the other place.”

“I think costs will go up because we'll just be using more tokens, not because the prices will necessarily go up, but then I think prices will come down because all of these frontier models are gonna be in a bit of a pricing war.”

“Everyone's starting to recognize that there might be a world in which like, you don't want to have the frontier models because you're in charge of the cost of that. And the returns are going to be more and more diminishing over time now.”

“Part of the driver of the difference is probably the willingness of Chinese companies to do scale distillation of the frontier models.”

“And I think you'll end up with companies using both mixing and matching them depending on the task at hand.”

“There's a growing portion of tasks in the economy and within these enterprises where you just don't need a frontier model, and oftentimes you don't even need anything close to a frontier model.”

“I think the thing you should anticipate is that over a year long time horizon to get to today's level of intelligence that feels very premier, it's going to be much cheaper. But there's going to be a new thing that is going to be so much better.”

“The open source models are about three to three to six months behind the frontier models, right? But they're 99% cheaper for inference.”

“So I, my guess is that within 12 to 18 months, 80% of our workloads will be going toward models that are 99, 99% cheaper, but 20% of it will still go to these frontier models where you need to be, you know, IQ maxing or whatever you want to call it.”

“And we got this thing up and running and out of the box, it's doing a 150 tokens per second, which is like 10 times what you get out of frontier models.”

“I was testing it, doing the exact same coding tasks that I'm doing with frontier models. It's like as good or better.”

“I mean, basically because the frontier models are that good, like, you know, I think that in two years, like, That'll be a 100,000 and, you know, down to 10,000 and a thousand. I mean, basically these orders of magnitude are about to happen.”

“Our team built this document sort of vision model that takes a page and gives you back a nice JSON with all the components. And it's very competitive. It's like probably like a hundred X cheaper than those frontier models and still better.”

“So generally speaking, the issue with this is that frontier models are extremely bad at automated red teaming because they have a lot of safeguards built into them. So if you try to use them to jailbreak another model, they will actually refuse their safety tr…”

“Right now, the frontier models know this problem, and they're aggressively moving to incorporate memory and context into their models because they understand that's the mode. And the challenge is you have to pay for it twice. If you say, no, I don't want to us…”

“And this was the kind of major unlock for the company as well, is we had all this massive data. We were able to post train on all of that user signal and now beat the frontier.”

“For our coding agents and, like, for actually developing product, we probably always want the frontier model, but if you're just, Automating a workflow. If the open source one that's basically free can automate it, then I don't, there's diminishing returns to …”

“We had started benchmarking frontier models on physics and science capabilities, and they were not very good. They were good at, like, doing things like summarization of papers, but if you said, hey, could you, like, analyze the scientific data coming out of a…”

“The current series of frontier models are largely in the trillion parameter realm. Yeah. Now we are starting to see 10 trillion parameter models.”

“I mean, I have a system at Turing that helps me run Turing, and this is a system that I built on top of one of the frontier models, and I didn't have to fine tune it. Like, all I had to do was intelligent management of context, memory, access to the right tool…”