Dec 31, 2025 · 24m · latent-space
[State of Evals] LMArena's $1.7B Vision — Anastasios Angelopoulos, LMArena
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this interview, Arena co-founder Anastasios Angelopoulos discusses the platform's evolution from a Berkeley research initiative into a venture-backed evaluation leader, detailing its organic benchmarking methodology, technical scaling, and multimodal expansion. He explains how Arena maintains platform integrity and subsidizes millions of blind model battles to establish an impartial North Star for frontier AI evaluation.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. The hosts hold 29.2% of the talking time here. How this is scored →
The hosts as informed peer 4.8 Guest teaching 4.0 Guest disagreement 2.2 The hosts pushing back 2.8
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 10:30 Dismissing the Leaderboard Illusion paper as unscientific
Anastasios directly dismisses the Cohere research paper critiquing LMSYS as fundamentally flawed and unscientific, challenging the motivations and competence behind its claims.
Hardest push from the hosts ▶ 8:03 Swyx defending pre-generated prompt evaluationsSwyx challenges Anastasios's dismissive stance on pre-rendered video arenas by arguing that users genuinely need to see outside examples to learn effective prompting.
Biggest teaching moment ▶ 11:40 Dissecting the data errors in Cohere's critiqueAnastasios educates the hosts on the exact factual errors published in the critique, explaining the true 60/40 open-to-closed sampling distribution and the community purpose of blind preview testing.
The host holds their own ▶ 15:04 Swyx showcasing multimodal paper diagram generationSwyx demonstrates his deep technical and content workflow expertise by detailing how he feeds dense RL research papers into Nano Banana Pro to produce PhD-level diagrams instantly.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Rebranding LMSYS to ARENA and Broadening Scope | 3 | 2 | 1 | 1 | The conversation starts warmly with light banter about branding, dropping 'LM' from LMSYS, and Ansh Sharma's early incubation of the company. Swyx and Alessio ask open-ended questions about the transition from Berkeley research project to a venture-backed startup. | |
| Deploying $100M and Measuring Organic User Scale | 5 | 4 | 2 | 4 | Swyx presses Anastasios on how ARENA will deploy $100M, questioning whether they pay enterprise inference rates and how they verify organic user distributions without mandatory logins. Anastasios explains their cost structure, user base scale, and survey methodology. | |
| Arena Methodology vs. Competitors and Prompt Engineering | 5 | 4 | 2 | 4 | The hosts bring up competing benchmark platforms like Artificial Analysis. Swyx defends the utility of pre-rendered prompts for users learning how to prompt, while Anastasios emphasizes ARENA's organic, user-driven query methodology. | |
| Technical Infrastructure: Migrating from Gradio to React | 6 | 6 | 4 | 2 | After briefly touching on migrating from Gradio to React, Swyx raises Cohere's 'Leaderboard Illusion' paper. Anastasios strongly critiques the paper as unscientific and refutes its claims regarding open-source sampling ratios and pre-release testing. | |
| The Economic and Practical Value of Multimodal Generation | 5 | 4 | 2 | 3 | Swyx discusses his initial skepticism toward image generation before highlighting real-world workflows with Nano Banana Pro. Anastasios outlines ARENA's non-negotiable integrity standards, insisting the public leaderboard operates strictly as an impartial loss leader. | |
| Strategic Product Focus, Community Growth, and User Retention | 5 | 4 | 2 | 3 | The discussion turns to product roadmap, resisting API feature creep, and consumer retention dynamics. Swyx pushes on the need to evaluate agentic software harnesses like Devin rather than raw foundational models alone. |
Statements from this episode (18)
Angelopoulos: Arena Dropped 'LM' to Broaden Beyond Language Models
“So, so we wanted to maybe broaden a little bit. And we were the first Serena, so we feel like let's kind of try to own that.”
Angelopoulos: Arena received grants from Sequoia and a16z before incorporating
“He was not, you know, A-sixteen was not the only one to do this. We also had a great grant from Sequoia, but Ansh was in particular quite, quite supportive of us and, you know, gave us some resources in order to continue building out Arena before we even We're…”
LMArena funds all model inference running on its platform
“We fund all of the inference on the platform.”
Angelopoulos: LMArena receives only standard enterprise inference discounts
“No, no, we get discounts, but they're, but they are standard enterprise discounts. The same that would be given to any other customer.”
Arena processes tens of millions of conversations monthly, totaling 250 million
“We have probably two hundred and fifty million conversations that happen over the course of the platform. We're on the order of, you know, mid tens of millions of conversations every month that are happening on the platform.”
Angelopoulos: 25% of LMArena platform users write software for a living
“25% of the people on our platform, for example, do software for a living.”
Angelopoulos: About half of LMArena users are authenticated
“About half of our users now are login.”
Arena's organic user prompts provide realism that Artificial Analysis lacks
“They have arenas, but the arenas are not based on organic usage. Like the thing that distinguishes our platform versus theirs is that the users are actually inputting their own use case. They're actually asking their own question. And that gives a level of rea…”
swyx: Artificial Analysis video arena uses pre-generated videos instead of user inputs
“So like, for example, for AA, their video arena is pre-generated videos. You can't enter in your own video.”
Gradio scaled Arena to 1 million monthly active users before migration
“Gradio scaled us to a million Mal.”
Angelopoulos: LMArena's top expenses are free-tier inference, hiring, and SF office
“Primarily inference that funds the free usage of the platform and then also hiring, of course, headcount. We have an office, you know. That's an SF.”
Arena sampled open-source models at 60/40, debunking Leaderboard Illusion paper
“But, you know, there, for example said that we were, that we only sampled, like, nine percent open source models and, like, you know, 60%, like, closed source models, and this created a gap between open and closed source. But in reality, we're actually really …”
Arena's anonymous Nano Banana test moved Google's stock and product roadmap
“I mean, that moment alone changed Google's like roadmap. Market share. Seriously. I mean, Google stock, billions of dollars are moving because of Nano.”
Angelopoulos: Academic paper figures will soon be generated by AI models
“Soon we're not going to be even making them for our papers. We're, they're just going to be, our paper figures are going to be made by Emily.”
Angelopoulos: LMArena has released more real-world AI data than almost anyone
“We've probably released more data than basically anybody on the real world use cases of AI.”
Arena's public leaderboard will never adopt Gartner-style pay-to-play models
“You can't pay to get on the public leaderboard. It's not like a Gartner in that sense. It's not like any of these, like you know, pay to play systems, never going to be like that. Models are going to be listed on the leaderboard, whether or not the providers p…”