Opinion
Mohan: Bearish on outsourced eval startups because AI companies must own evaluations
“And I guess maybe one of the things I'm a little bearish on is If another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing? What are they really doing at that point? If t…”
What-if
Varun Mohan: Codeium would have failed if it used vLLM
“If we use VLLM, we would not be talking with you right now.”
Insight
Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. Bu…”
Assertion Not checkable as stated
Mohan: Codeium quality matches Copilot and drives user churn
“The product is actually one of those products where even use Copilot and use us, it's hard to tell the difference actually. And a lot of our users have actually churned off of Copilot.”
Insight
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Opinion
Mohan: Windsurf's real value is large codebase work, not 0-to-1 apps
“We had the technology to go out and build these zero to one apps very quickly, and I think people are using Windsurf to actually do that, and it's like extremely impressive, but the real value, I think, is actually much deeper than that. It's actually that you…”
Insight
Mohan: Squeezing the last 10% from AI benchmarks is counterproductive
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really op…”
Prediction Not checkable as stated
Mohan: Explicit user prompting will soon become an anti-pattern in AI coding
“I actually think asking people to do things explicitly is probably going to be more of an anti-pattern if we can actually go and passively suggest the entire change for the user.”
Disclosure
Mohan: Codeium uses in-house models for autocomplete due to poor frontier FIM
“The things like autocomplete and supercomplete that run on every keystroke are entirely, like, our own models, and by the way, that is still because properties like FIM, fill in the middle capabilities are still quite bad with the current model.”
Insight
Mohan: Vector embeddings fail at granular code retrieval queries
“For a lot of the systems, we do believe embeddings work, but for complex questions, We don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a code base of find me all quadratic time algo…”
Insight
Mohan: Monetizing individual developers in AI tooling is the wrong strategy
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Assertion Not checkable as stated
Mohan: Codeium generated dynamic PNGs due to VS Code API limitations
“On VS Code, actually, the problem for us wasn't actually being able to implement the feature. We had the feature for a while. Problem was actually even to show the feature, VS Code would not expose an API for us to do this. So what we actually ended up doing w…”
Insight
Mohan: Monetizing individual developers is flawed due to low switching costs
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Insight
Mohan: AI coding tools create a self-fulfilling loop by changing developer behavior
“Once you start using products like this, where in the beginning there's like skepticism, like how, how valuable can it be? And suddenly now like user behavior fundamentally changes so that now when I need to write a function, I'm like documenting my code more …”
Insight
Mohan: AI startups should bootstrap on existing APIs unless fine-tuning on large datasets
“Unless you are, you probably should always bootstrap on top of an existing API, right? Because like, even if you were to, the only reason why we didn't is because we knew that this product was actually buildable. Probably if we worked hard enough to train a mo…”
Insight
Mohan: LLM products depend on latency, quality, and correctability
“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality…”
Disclosure
Mohan: Codeium trains smaller, compute-suboptimal models for lower inference latency
“And for our case, our models that we train internally, they might not be the most compute optimal. In other words, we probably could have had a better model by making it larger, but the trade-off would have been latency. We know what the impact of having highe…”
Opinion
Mohan: Current vector databases are unviable for terabyte-scale retrieval
“I'm actually not convinced that the current breed of embedding database companies are like ready for what the future sort of looks like, because I think I'm just looking at their pricing, how much it costs per gigabyte, and it's prohibitive at the scale we're …”
Assertion Not checkable as stated
Mohan: Codeium bypassed VS Code API limits by rendering Supercomplete via PNGs
“Problem was actually even to show the feature VS code would not expose an API for us to do this. So what we actually ended up doing was dynamically generating PNGs to actually go out and showcase this.”
Insight
Mohan: Stripping open-source commits turns coding agent evals into a continuous problem
“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”
Insight
Mohan: Developers will never completely articulate problem statements to AI
“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”
Insight
Mohan: Senior engineers gained little value from ChatGPT due to context friction
“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight yea…”
Insight
Mohan: Buying third-party software risks permanently losing core engineering competencies
“When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like, startups are so limited on time.”
What-if
Mohan: Codeium would not have survived if it relied on vLLM
“If we use VLLM, we would not be talking with you right now.”
