Vercel CTO Malte Ubl reacts to Anthropic's Boris Power claiming Claude's coding agent business was built on vibes rather than offline evaluations.
Insight
Malte Ubl: When Vibes and Eval Data Disagree, Vibes Are Right
“I think that the common quip that if the vibes and the data disagree, the vibes are probably right. It's true, right? So you have to like, be honest with yourself, like, do they agree and kind of iterate On them over time.”
Disclosure
Ubl: Vercel's Composite Models Are Faster Than Agentic Loops
“Basically what we do is we have this like composite model architecture. We run the frontier model and then we run the fine tune model after to fix its errors. That doesn't perform better than an agentic loop, but it's orders of magnitude faster, right?”
Opinion
Ubl: Vercel leads the truly open-source deployment business model
“Vercel maybe has not invented this, but certainly kind of is the most successful at a model where you say, okay, I have this software library and it's truly open source. Everyone can run it. It comes with like adapters for every place on the planet, and that m…”
Prediction Not checkable as stated
Future app platforms must extract auth and authorization completely
“Auth cannot be part of the app, because they're not going to get that right, right? So, Auth has to be extracted from the app. In fact, which data you can see, they also cannot be under control of the app, because again, you're going to get it wrong, right? So…”
Disclosure
Ubl: Vercel publishes evals to influence OpenAI and Anthropic models
“I'm Vercel and I publish at Eval. That I want OpenAI and Anthropic to use to make sure when they ship the next model that they're better at the stuff that I care about.”
Insight
Thick abstractions in nascent AI frameworks are usually wrong
“And so if you put a very thick abstraction, then it's probably going to be the wrong abstraction. You have to, so you have to be humble and say, okay, I need to stay low level. So that this can be flexibly used as trends emerge, right?”