The Ledger, every show

“And for the first time ever, we saw a couple of weeks ago that Kimi K three actually beat the best closed source American models on a, you know, pretty important subset of tasks, for example, front end coding, like web development, which a huge fraction of dev…”

“If you look at the whole space of all inference, most of it is still being consumed on first party APIs and on proprietary models. That's why anthropic revenue has been just a total hockey stick. It's not, you know, it's not like they're being completely canni…”

“People don't know this, but Arena's one of the largest consumer AI apps in the world. We're bigger than, like, XAI. We're bigger than, like, Hugging Face, and Manus, and GenSpark, where it's so massive, like if you, like outside in, it's like 30 plus million m…”

“But, you know, there, for example said that we were, that we only sampled, like, nine percent open source models and, like, you know, 60%, like, closed source models, and this created a gap between open and closed source. But in reality, we're actually really …”

“That's why we're making style control default.”

“When you train a prompt to leaderboard model, which is like, let's say a seven billion parameter model, and then you use it to route on just questions on the arena and everybody's questions, that model does better than any of the constituent models that were u…”

“Now, if you trace the performance, the best performance that, you know, any individual model can give you as part of the router as a function of cost. That's like two X worse than the router. In other words, the router is giving you double the bang for your bu…”

“So there's this model and it crushed the benchmark. You know, it's just like really like a big gap. And what that's telling us is that it's not saturated yet. And so it's still measuring some signal that was encouraging point.”

“None of the leaderboard currently is apples to apples, because you have, like, Gemini Flash, you have, you know, all sorts of tiny models, like Llama Like, eight B and four or five B are not apples to apples.”

“So they're way hardware constrained over there. And they've been trying to like black market import chips because of this. And you see this in the news, right? The information just reported on this.”

“Gemma, by the way, is pretty good in terms of efficiency. If you look at arena, you'll see the, on the Pareto curves of like performance versus cost. Gemma's on there.”

“Harvey, the CEO of Harvey himself is saying that, you know, his biggest competitive worry is the model labs.”

“Okay, let's move from Elo to Bradley Terry because we're actually performing an estimate here instead of just like You know, and the ELO score moves over time. It doesn't converge, but Rally Terry models converge and how do we then construct confidence interva…”

“We're going to keep publishing papers. We're going to keep releasing open source. We're going to keep releasing open data.”

“He was not, you know, A-sixteen was not the only one to do this. We also had a great grant from Sequoia, but Ansh was in particular quite, quite supportive of us and, you know, gave us some resources in order to continue building out Arena before we even We're…”

“Gradio scaled us to a million Mal.”

“The arena score that we show on our leaderboard is a particular type of linear model, right? It's a linear model that takes, it's a logistic regression that takes model identities and fits them against human preference, right? So it regresses human preference …”