Assertion Not checkable as stated
Chen: Researchers Degrade AI Factuality Just to Boost LMSYS Rankings
“A lot of researchers, they'll tell us that their VPs make them focus on increasing their rank on LMSYS. And so I've had researchers explicitly tell me that they're okay with making their models worse. Add factuality works at following instructions as long as i…”
Opinion
Chen: Many YC Founders Only Care About Fundraising Brags and Headlines
“Like, if you talk to a bunch of YC founders or whoever it is, like, what is their goal? It really is to tell all their friends that they raised ten million dollars and show their parents they got a headline on TechCrunch. Like, that is their goal.”
Opinion
Chen: LMSYS Chatbot Arena is a giant plague on AI
“One of the things I think is a giant plague on AI is Elimsis, Elimarena.”
Assertion Not checkable as stated
Chen: Surge AI Is Largest Human Data Player in AI
“We are kind of like the biggest human data player in this space.”
Opinion
Chen: Most AI Data Competitors Are Non-Technical Body Shops
“A lot of other companies in this space, they are essentially just body shops. What they are delivering is not data. They are literally just delivering warm bodies to companies. And so what that means is like at the end of the day, they don't have any technolog…”
Opinion
Chen: Meta's Deal With Scale AI Actually Benefited Surge AI
“It's been beneficial because yeah, there were still some legacy teams using scale. Like they just didn't know about us because we were still pretty under the radar.”
Assertion Not checkable as stated
Chen: The Easiest Way to Boost LMSYS Rank Is Longer Responses
“Like one of the things that we found is that the easiest way to improve your rank on album arena is the ability to make your model response longer.”
Insight
Chen: Early Startups Should Not Hire Data Scientists as First Employees
“Like I would never hire data scientists when the first three people in a company. And I say that because I used to be a data scientist. Like data scientists are great when you want to optimize your product by two percent or five percent, but that's definitely …”
Insight
Chen: Early-stage startups should not hire product managers
“Product managers are great when your company gets big enough. But at the beginning, you should be thinking about yourself about what product you want to build. And your engineer should be hands-on. They should be having great ideas as well.”
Insight
Chen: AI RL Environments Are Too Complex to Create Synthetically
“I think one of the things that people really underestimate is how it is, how complicated it is that you can't just synthetically generate it.”
Opinion
Chen Bets on xAI to Catch Up With Top Frontier Labs
“So I would bet on XAI. I think they're just very hungry and mission-oriented in a way that gives them a lot of really unique advantages.”
Prediction Not checkable as stated
Chen: Frontier AI Models Will Proliferate Rather Than Commoditize
“So I actually see more and more frontier models opening up over time because I actually don't think that the models will be commodities.”
Assertion Not checkable as stated
Chen: Surge AI Operates With Just Over 100 Employees
“And we're about a hundred, a little over a hundred people.”
Insight
Chen: Generative AI Data Has an Almost Unlimited Quality Ceiling
“So there's a very, very low ceiling on the bar of quality, but then take something like writing poetry. Well, I suck at writing poetry. Hemingway's definitely want to write a much better poem than I am. Or imagine, I don't know, a VC pitch deck. You're going t…”
Prediction Not checkable as stated
Chen: Synthetic RL Environments Alone Won't Meet Future Frontier AI Demand
“Like, I don't think our environments alone will suffice just because, I mean, it depends on how you think about our environments, but oftentimes these are very, very rich trajectories are very, very long. And so it's almost like inconceivable that a single rew…”
Disclosure
Chen: Surge AI Heavily Uses Synthetic Data to Supplement Human Labelers
“Like we use it like a ton ourselves in order to supplement what the humans do.”
Disclosure
Chen: Surge AI Works With All Frontier Labs on Model Evaluation
“So internally we do a lot of work actually today with working with all the frontier labs to help them understand their models. So again, we're constantly evaluating them. We're constantly surfacing loss areas for them to improve on and so on and so on.”
Assertion Not checkable as stated
Chen: Frontier AI Labs Are No Longer Publishing Research
“Like, I think it is really interesting in that a lot of the, like for obvious reasons, a lot of the frontier labs, they're just not publishing anymore.”
Disclosure
Chen: Surge AI Was Profitable From Day One, Avoiding Venture Capital
“I think we were very, very lucky to be profitable from the start. So we didn't need the money. It always felt weird to give up control.”
