The Ledger, every show

“They come in, they're like, hey, I want to be an infrastructure engineer at Arena, which is a great job that we're hiring for you. But then the other side of it is some guy that looks perfectly normal. They're getting, you know, they're passing all of our tech…”

“And I think that the reason to be worried is because if the open source ecosystem somehow makes the cost saving opportunity for businesses much more salient and therefore decreases the revenue of companies like OpenAI and Anthropic within the enterprise, that …”

“And for the first time ever, we saw a couple of weeks ago that Kimi K three actually beat the best closed source American models on a, you know, pretty important subset of tasks, for example, front end coding, like web development, which a huge fraction of dev…”

“If you look at the whole space of all inference, most of it is still being consumed on first party APIs and on proprietary models. That's why anthropic revenue has been just a total hockey stick. It's not, you know, it's not like they're being completely canni…”

“Software is no longer really a moat because it can be produced instantaneously, right? Let's project out five years. That's what's going to happen. And so what moats exist? Network effects exist and data moats exist.”

“I believe that we're going to have at least one massive, you know, multi-hundred billion, if not trillion dollar American company focused on American first open source.”

“And within that time, they'd become the number one American open source model. But then the less generous take would be the companies existed for a year and a half. And then they've come up with, yes, the number one American open source, but there's nine Chine…”

“Yeah, my guess my guess would be that we will. I, I'm not saying I support it, but I think that it is likely where the world is headed. If I had to like place a bet, it would be there, but I think it's very uncertain at the moment.”

“For example, one of the things that's going to happen is that like right now, Anthropic has like disgustingly high growth, gross margins in their inference.”

“And after they go public, the whole world is going to see that, right? Like we're going to see their margins because those are going to be public information. And that's going to exert downward pricing pressure on their inference.”

“So I think we need guardian models and also, you know, agents within our businesses. What is a guardian model? Something that can witness the traces basically that's looking over the shoulder of every agent within a business and then saying, okay, this is a sa…”

“There's at least 75 Neolabs. And For sure, like two thirds of those are going to be worth nothing or like they're going to be bought out for parts, right? That's going to be like an aqua hire.”

“I believe it's going to be at least a hundred billion dollars by 2030, if not a trillion.”

“People don't know this, but Arena's one of the largest consumer AI apps in the world. We're bigger than, like, XAI. We're bigger than, like, Hugging Face, and Manus, and GenSpark, where it's so massive, like if you, like outside in, it's like 30 plus million m…”

“I think it's hard to say not NVIDIA. I think NVIDIA is probably in the lead there.”

“But I think the enterprise adoption of AI is going to be another 10 Xer for the industry. I think it'll 10 X NVIDIA very reliably.”

“I mean, that moment alone changed Google's like roadmap. Market share. Seriously. I mean, Google stock, billions of dollars are moving because of Nano.”

“Static benchmarks overfit. Why? It's because as Jan said earlier, you're giving the student the same test over and over. You have a model, you test it, you know, you look at whether or not it's improved on a static data set. Then you find another model, you te…”

“I don't really see them banning the use of Chinese models in the U.S. I don't think it makes sense for them.”

“It's not only true that they're terrified of working with the frontier labs, but they're also terrified of working with the Chinese open source.”

“Companies are spending on it, usually within Frontier Labs, at about 10 to 20% about the amount that they're spending on GPUs.”

“I think that the, like, level of just human flourishing that's going to happen as we start to one by one eradicate diseases the same way that we're currently eradicating open problems in math is going to be incredible.”

“That's exactly one of the areas where the data layer, where you can clearly see that the data layer is where value is going to accrue. Because the GPUs Are the same GPUs in both cases. The problem is that, that data infrastructure, the flywheel, the data colle…”

“But, you know, there, for example said that we were, that we only sampled, like, nine percent open source models and, like, you know, 60%, like, closed source models, and this created a gap between open and closed source. But in reality, we're actually really …”

“That's why we're making style control default.”

“Absolutely. Absolutely. It should be personalized just for you. You should understand which models are best for you.”

“When you train a prompt to leaderboard model, which is like, let's say a seven billion parameter model, and then you use it to route on just questions on the arena and everybody's questions, that model does better than any of the constituent models that were u…”

“Now, if you trace the performance, the best performance that, you know, any individual model can give you as part of the router as a function of cost. That's like two X worse than the router. In other words, the router is giving you double the bang for your bu…”

“So there's this model and it crushed the benchmark. You know, it's just like really like a big gap. And what that's telling us is that it's not saturated yet. And so it's still measuring some signal that was encouraging point.”

“None of the leaderboard currently is apples to apples, because you have, like, Gemini Flash, you have, you know, all sorts of tiny models, like Llama Like, eight B and four or five B are not apples to apples.”

“So they're way hardware constrained over there. And they've been trying to like black market import chips because of this. And you see this in the news, right? The information just reported on this.”

“And by the way, it's worth noting that China has already restricted the use of American models within China, right? So if you look at the two by two matrix of US China restrict, not restrict, you know, like export import stuff They have already restricted the …”

“Gemma, by the way, is pretty good in terms of efficiency. If you look at arena, you'll see the, on the Pareto curves of like performance versus cost. Gemma's on there.”

“Harvey, the CEO of Harvey himself is saying that, you know, his biggest competitive worry is the model labs.”

“We have probably two hundred and fifty million conversations that happen over the course of the platform. We're on the order of, you know, mid tens of millions of conversations every month that are happening on the platform.”

“Soon we're not going to be even making them for our papers. We're, they're just going to be, our paper figures are going to be made by Emily.”

“We've probably released more data than basically anybody on the real world use cases of AI.”

“It's true that people vote for longer responses, you know, preferentially over shorter responses, even given the same contents or well-known human bias.”

“We don't claim to be an AGI benchmark. We are faithfully representing the preferences of our community.”

“Okay, let's move from Elo to Bradley Terry because we're actually performing an estimate here instead of just like You know, and the ELO score moves over time. It doesn't converge, but Rally Terry models converge and how do we then construct confidence interva…”

“A lot of people don't know this, but ShopBot Arena is Used by like a million plus monthly users. We get like, you know, tens of thousands of votes on a daily basis. We have like over like, you know, a hundred fifty million conversations that have been had on t…”

“We're going to keep publishing papers. We're going to keep releasing open source. We're going to keep releasing open data.”

“The fundamental is organic, real-world testing with feedback. That's not going to change. I can tell you that that is not going to change.”

“We don't do that right now, partially because we kind of have know from simulations that the amount of selection bias you incur with these five things is just not huge. It's not huge in comparison to the variability that you get from the, from just regular hum…”

“He was not, you know, A-sixteen was not the only one to do this. We also had a great grant from Sequoia, but Ansh was in particular quite, quite supportive of us and, you know, gave us some resources in order to continue building out Arena before we even We're…”

“We fund all of the inference on the platform.”

“25% of the people on our platform, for example, do software for a living.”

“About half of our users now are login.”