Chiang: Online dynamic benchmarks are slower and more expensive than offline benchmarks
Wei-Lin Chiang · In the Arena: How LMSys changed LLM Benchmarking Forever · Nov 1, 2024 · at 7:48
LMSYS co-founder Wei-Lin Chiang discusses trade-offs between dynamic human preference arenas and static offline benchmarks in large language model development.
0:00 / 0:11exact quote · 11.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“This kind of like online dynamic benchmark is slow, is more expensive than Standing benchmark, offline benchmark, where people still need it, like when they build models, they need static benchmark to track.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Wei-Lin Chiang
Chiang: Vicuna Showed Open-Weight Models Could Match ChatGPT Quality
“And people were very excited about it because it kind of like demonstrate open way model can reach this conversation capability similar to ChatGPT.”
Chiang: There are currently no good benchmarks for evaluating LLM routers
“Right now, currently, there seems to be the, one of the end point when we developed this project was like, there's just no good benchmark for a router.”