Assertion certainty 4/5 debate potential 1/5

Chiang: Coding questions drive 20% to 30% of Chatbot Arena usage

Wei-Lin Chiang · In the Arena: How LMSys changed LLM Benchmarking Forever · Nov 1, 2024 · at 10:36

LMSYS co-founder Wei-Lin Chiang describes user demographic skew on the Chatbot Arena LLM evaluation platform.

0:00 / 0:05exact quote · 5.5s

▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark

“We do see a lot of like developers come to the site asking polling questions. Only 30%.”

quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →

The claim, plainly

Roughly 20% to 30% of users visiting Chatbot Arena are developers asking coding questions.

a quote alone can be ambiguous, so this restates it with the references and context of the conversation around it resolved. AI-written; the tape below the player is the source.

How this resolves

Verification couldn't pin this to a public evidence class, so it stays on the record with its date, unassessed rather than guessed.

not yet assessed: in the resolution queue (grounded fact checks run in waves; this one's next)

Mentioned

Chatbot Arena developer LMSYS

⚑ Spot an error? Report it

More from Wei-Lin Chiang

Assertion Not checkable as stated

Chiang: Vicuna Showed Open-Weight Models Could Match ChatGPT Quality

“And people were very excited about it because it kind of like demonstrate open way model can reach this conversation capability similar to ChatGPT.”

Wei-Lin Chiang Nov 1, 2024 ▶ 3:01 In the Arena: How LMSys changed LLM Benchmarking Forever

Assertion Not checkable as stated

Chiang: There are currently no good benchmarks for evaluating LLM routers

“Right now, currently, there seems to be the, one of the end point when we developed this project was like, there's just no good benchmark for a router.”

Wei-Lin Chiang Nov 1, 2024 ▶ 35:57 In the Arena: How LMSys changed LLM Benchmarking Forever

Insight

Chiang: Online dynamic benchmarks are slower and more expensive than offline benchmarks

“This kind of like online dynamic benchmark is slow, is more expensive than Standing benchmark, offline benchmark, where people still need it, like when they build models, they need static benchmark to track.”

Wei-Lin Chiang Nov 1, 2024 ▶ 7:48 In the Arena: How LMSys changed LLM Benchmarking Forever

Assertion Not checkable as stated

Chiang: Chatbot Arena almost died after launch due to low engagement

“At some point, almost died. Because as you can imagine, this leaderboard depends on user, like part of, like community engagement participation. If no one comes to vote, Tomorrow then no deal.”

Wei-Lin Chiang Nov 1, 2024 ▶ 11:10 In the Arena: How LMSys changed LLM Benchmarking Forever

Chiang: Coding questions drive 20% to 30% of Chatbot Arena usage

More from Wei-Lin Chiang

Turn any episode into a week of clips.