The Ledger, every show

“That doesn't mean that they're not distilling. They may still be using distillation as a sub-step in their training procedure, But it does mean that distillation is only part of the story and that there's something that those labs are doing above and beyond di…”

“They come in, they're like, hey, I want to be an infrastructure engineer at Arena, which is a great job that we're hiring for you. But then the other side of it is some guy that looks perfectly normal. They're getting, you know, they're passing all of our tech…”

“The first is that I think that Silicon Valley investors have become total bitches with respect to revenue concentration. It's like, what the, what are you talking about? Like TSMC has revenue concentration. There's businesses that are like many hundreds of bil…”

“And I think that the reason to be worried is because if the open source ecosystem somehow makes the cost saving opportunity for businesses much more salient and therefore decreases the revenue of companies like OpenAI and Anthropic within the enterprise, that …”

“I have to say, I also just like completely disagree with the foundation of the question. The like implicit assumption is that like chat is easy or that it's even easier than web dev. That's completely false. It's a completely naive perspective that people have…”

“And for the first time ever, we saw a couple of weeks ago that Kimi K three actually beat the best closed source American models on a, you know, pretty important subset of tasks, for example, front end coding, like web development, which a huge fraction of dev…”

“If you look at the whole space of all inference, most of it is still being consumed on first party APIs and on proprietary models. That's why anthropic revenue has been just a total hockey stick. It's not, you know, it's not like they're being completely canni…”

“Software is no longer really a moat because it can be produced instantaneously, right? Let's project out five years. That's what's going to happen. And so what moats exist? Network effects exist and data moats exist.”

“I believe that we're going to have at least one massive, you know, multi-hundred billion, if not trillion dollar American company focused on American first open source.”

“And within that time, they'd become the number one American open source model. But then the less generous take would be the companies existed for a year and a half. And then they've come up with, yes, the number one American open source, but there's nine Chine…”

“What if the other side that's interacting with the chat bot can, you know, build in a certain code word or a certain like character sequence that then jail breaks that model and gets it to reveal all the data to me. So it can sort of like vomit out all of the …”

“Yeah, my guess my guess would be that we will. I, I'm not saying I support it, but I think that it is likely where the world is headed. If I had to like place a bet, it would be there, but I think it's very uncertain at the moment.”

“For example, one of the things that's going to happen is that like right now, Anthropic has like disgustingly high growth, gross margins in their inference.”

“And after they go public, the whole world is going to see that, right? Like we're going to see their margins because those are going to be public information. And that's going to exert downward pricing pressure on their inference.”

“So I think we need guardian models and also, you know, agents within our businesses. What is a guardian model? Something that can witness the traces basically that's looking over the shoulder of every agent within a business and then saying, okay, this is a sa…”

“There's at least 75 Neolabs. And For sure, like two thirds of those are going to be worth nothing or like they're going to be bought out for parts, right? That's going to be like an aqua hire.”

“And the thing that you really need to believe is that if the valuation is ten billion today, that you're going to generate the revenue, let's say it's a 30 X revenue multiple or 25 X revenue multiple to become a hundred billion dollar business. And so what tha…”

“I believe it's going to be at least a hundred billion dollars by 2030, if not a trillion.”

“People don't know this, but Arena's one of the largest consumer AI apps in the world. We're bigger than, like, XAI. We're bigger than, like, Hugging Face, and Manus, and GenSpark, where it's so massive, like if you, like outside in, it's like 30 plus million m…”

“If, like, inference is going to commoditize, then, of course, the next best thing is for the model providers to be moving up the application layer in order to own more of the application stack so that they ensure that they're not commoditized and they're getti…”

“I think it's hard to say not NVIDIA. I think NVIDIA is probably in the lead there.”

“But I think the enterprise adoption of AI is going to be another 10 Xer for the industry. I think it'll 10 X NVIDIA very reliably.”

“I mean, that moment alone changed Google's like roadmap. Market share. Seriously. I mean, Google stock, billions of dollars are moving because of Nano.”

“In reality, even in such industries, the majority of questions that people ask are subjective. Okay. So the mythology that in hard sciences or in mission critical industries, people just have like cut and dried questions and they just need like a retrieval and…”

“Static benchmarks overfit. Why? It's because as Jan said earlier, you're giving the student the same test over and over. You have a model, you test it, you know, you look at whether or not it's improved on a static data set. Then you find another model, you te…”

“Static benchmarks are intrinsically, to some extent, unable to measure generative model performance. And the reason is because you cannot Pre-annotate all the outputs of a generative model. You change the model. It's like the distribution of your data is chang…”

“I do think the combination of FDE plus Open American model May be a more sustainable model for the future of American or even Western businesses, because it, because they might not want to be building on top of external third party services. They might want to…”

“The downside of export control is that it can incentivize them to build their own ecosystem, and then what do we do?”

“I don't really see them banning the use of Chinese models in the U.S. I don't think it makes sense for them.”

“It's not only true that they're terrified of working with the frontier labs, but they're also terrified of working with the Chinese open source.”

“We're gonna change our whole hiring process because of this kind of stuff. It absolutely worries me. Well, at first you need to verify that person is real. So all of our onboarding, we're considering at least making all of our onboarding in person because of t…”

“Companies are spending on it, usually within Frontier Labs, at about 10 to 20% about the amount that they're spending on GPUs.”

“The data is really the hardest part of model training. Because you need to source it. It's so dirty. Nobody wants to do that shit. Nobody wants to hire all these people to generate data and then, you know, turn that into basically data plus GPUs equals model. …”

“So we're past a hundred million in annualized revenue run rate, and that's based on like Q, Q two times four.”

“I think that the SaaS-pocalypse has been a little bit overstated overall. Because people don't understand always the dynamics of those businesses and how tough it is to replicate what they've built just also from a network perspective and a data perspective.”

“It's like priority number 12 for Anthropic is probably not high enough for Harvey and LaGuardia to be too scared.”

“I think that the, like, level of just human flourishing that's going to happen as we start to one by one eradicate diseases the same way that we're currently eradicating open problems in math is going to be incredible.”

“That's exactly one of the areas where the data layer, where you can clearly see that the data layer is where value is going to accrue. Because the GPUs Are the same GPUs in both cases. The problem is that, that data infrastructure, the flywheel, the data colle…”

“They have arenas, but the arenas are not based on organic usage. Like the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question. And that gives a level of rea…”

“But, you know, there, for example said that we were, that we only sampled, like, nine percent open source models and, like, you know, 60%, like, closed source models, and this created a gap between open and closed source. But in reality, we're actually really …”

“You can't pay to get on the public leaderboard. It's not like a Gartner in that sense. It's not like any of these, like you know, pay to play systems, never going to be like that. Models are going to be listed on the leaderboard, whether or not the providers p…”

“Devin's not gone. Devin's everywhere.”

“That's why we're making style control default.”

“Absolutely. Absolutely. It should be personalized just for you. You should understand which models are best for you.”

“The fact that we come from Berkeley and from a university really speaks to our scientific approach in neutrality. I think if it came from an industrial lab, people would always have questions about, oh, well, these people are they also training a model and wha…”

“Because language models are sort of the intermediary that gets you to this evaluation, there's also a scaling law that comes along with it. Which is to say that the more data you get, the bigger you build the platform, the better you can make your evaluations,…”

“Pairwise comparison feedback is not the only kind of feedback that we can use to construct leaderboards. We can construct leaderboards with any form of feedback.”

“When you train a prompt to leaderboard model, which is like, let's say a seven billion parameter model, and then you use it to route on just questions on the arena and everybody's questions, that model does better than any of the constituent models that were u…”