Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
Varun Mohan · 🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R) · Jul 28, 2025 · at 14:18
Varun Mohan, co-founder and CEO of Codeium/Windsurf, discusses the challenges of LLM inference latency and batching efficiency for real-time code generation.
0:00 / 0:45exact quote · 45.8s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. But one of the weird things about generation models is they're like autoregressive, at least for the time being, they're autoregressive. So the latency for a generation is a function of the amount of tokens that you actually end up generating. Like that's like the math. And you can imagine while you're generating the tokens though, Unless you batch a lot, it's going to end up being the case that you're not going to get great flop utilization on the hardware. So there's like a bunch of trade-offs here where if you end up using something completely off the shelf, like one of these serving things, serving frameworks, you're going to end up leaving a lot of performance on the table.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Varun Mohan
Mohan: Bearish on outsourced eval startups because AI companies must own evaluations
“And I guess maybe one of the things I'm a little bearish on is If another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing? What are they really doing at that point? If t…”
Varun Mohan: Codeium would have failed if it used vLLM
“If we use VLLM, we would not be talking with you right now.”
Mohan: Codeium quality matches Copilot and drives user churn
“The product is actually one of those products where even use Copilot and use us, it's hard to tell the difference actually. And a lot of our users have actually churned off of Copilot.”
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Mohan: Windsurf's real value is large codebase work, not 0-to-1 apps
“We had the technology to go out and build these zero to one apps very quickly, and I think people are using Windsurf to actually do that, and it's like extremely impressive, but the real value, I think, is actually much deeper than that. It's actually that you…”
Mohan: Squeezing the last 10% from AI benchmarks is counterproductive
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really op…”