The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 76 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Mohan: Bearish on outsourced eval startups because AI companies must own evaluations
“And I guess maybe one of the things I'm a little bearish on is If another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing? What are they really doing at that point? If t…”
Varun Mohan: Codeium would have failed if it used vLLM
“If we use VLLM, we would not be talking with you right now.”
Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. Bu…”
Mohan: Codeium quality matches Copilot and drives user churn
“The product is actually one of those products where even use Copilot and use us, it's hard to tell the difference actually. And a lot of our users have actually churned off of Copilot.”
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Mohan: Windsurf's real value is large codebase work, not 0-to-1 apps
“We had the technology to go out and build these zero to one apps very quickly, and I think people are using Windsurf to actually do that, and it's like extremely impressive, but the real value, I think, is actually much deeper than that. It's actually that you…”
Mohan: Squeezing the last 10% from AI benchmarks is counterproductive
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really op…”
Mohan: Explicit user prompting will soon become an anti-pattern in AI coding
“I actually think asking people to do things explicitly is probably going to be more of an anti-pattern if we can actually go and passively suggest the entire change for the user.”
Mohan: Codeium uses in-house models for autocomplete due to poor frontier FIM
“The things like autocomplete and supercomplete that run on every keystroke are entirely, like, our own models, and by the way, that is still because properties like FIM, fill in the middle capabilities are still quite bad with the current model.”
Mohan: Vector embeddings fail at granular code retrieval queries
“For a lot of the systems, we do believe embeddings work, but for complex questions, We don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a code base of find me all quadratic time algo…”
Mohan: Monetizing individual developers in AI tooling is the wrong strategy
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Mohan: Codeium generated dynamic PNGs due to VS Code API limitations
“On VS Code, actually, the problem for us wasn't actually being able to implement the feature. We had the feature for a while. Problem was actually even to show the feature, VS Code would not expose an API for us to do this. So what we actually ended up doing w…”
Mohan: Monetizing individual developers is flawed due to low switching costs
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Mohan: AI coding tools create a self-fulfilling loop by changing developer behavior
“Once you start using products like this, where in the beginning there's like skepticism, like how, how valuable can it be? And suddenly now like user behavior fundamentally changes so that now when I need to write a function, I'm like documenting my code more …”
Mohan: AI startups should bootstrap on existing APIs unless fine-tuning on large datasets
“Unless you are, you probably should always bootstrap on top of an existing API, right? Because like, even if you were to, the only reason why we didn't is because we knew that this product was actually buildable. Probably if we worked hard enough to train a mo…”
Mohan: LLM products depend on latency, quality, and correctability
“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality…”
Mohan: Codeium trains smaller, compute-suboptimal models for lower inference latency
“And for our case, our models that we train internally, they might not be the most compute optimal. In other words, we probably could have had a better model by making it larger, but the trade-off would have been latency. We know what the impact of having highe…”
Mohan: Current vector databases are unviable for terabyte-scale retrieval
“I'm actually not convinced that the current breed of embedding database companies are like ready for what the future sort of looks like, because I think I'm just looking at their pricing, how much it costs per gigabyte, and it's prohibitive at the scale we're …”
Mohan: Codeium bypassed VS Code API limits by rendering Supercomplete via PNGs
“Problem was actually even to show the feature VS code would not expose an API for us to do this. So what we actually ended up doing was dynamically generating PNGs to actually go out and showcase this.”
Mohan: Stripping open-source commits turns coding agent evals into a continuous problem
“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”
Mohan: Developers will never completely articulate problem statements to AI
“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”
Mohan: Senior engineers gained little value from ChatGPT due to context friction
“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight yea…”
Mohan: Buying third-party software risks permanently losing core engineering competencies
“When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like, startups are so limited on time.”
Mohan: Codeium would not have survived if it relied on vLLM
“If we use VLLM, we would not be talking with you right now.”
Mohan: GitHub might have under 10% full penetration in Fortune 500
“GitHub might have less than 10% penetration of the Fortune 500, full penetration. It's very small.”
Mohan: Switching SCM tools is harder than switching databases
“GitHub has very high switching costs or source code management tools, right? Because you actually need to switch over all the dependent systems on this workflow software. It's much harder than even switching off of a database.”
Mohan: Developers will never provide complete problem statements to AI
“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”
Mohan: Squeezing the final 10% on AI benchmarks encourages p-hacking
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is like counterintuitive to the actual goal of what the benchmark was. Like you probably should find a new hill to climb rather than sort of p-hacking or really optimi…”
Mohan: Explicit user prompting will become an AI coding anti-pattern
“I actually think asking people to do things explicitly is probably going to be more of an anti-pattern if we can actually go and passively suggest the entire change for the user. So almost imagine as the user is using the product that we're going to suggest th…”
Mohan: Embeddings fail at complex codebase retrieval tasks like identifying quadratic algorithms
“For a lot of the systems, we do believe embeddings work, but for complex questions, we don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a codebase of find me all quadratic time algor…”
Mohan: Windsurf uses Claude for planning, proprietary models for retrieval and diffs
“No, so actually the way it works is the high-level planning that is going on in the model is actually getting done with products like the Cloud. But the extremely fast retrieval, as well as the ability to, like, take the high-level plan and actually apply it t…”
Mohan: Codeium builds proprietary indexing and serving rather than paying third parties
“We don't really like pay other providers to do things for indexing. We don't pay like other providers to do our serving of our own custom models, right? And I think that's a core competency within our company that we have decided to build, but that's also enab…”
Mohan: Parallel multi-agent coding trajectories do not require separate remote machines
“If you want to run all of them in parallel, you probably don't want any machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writing ou…”
Varun Mohan: Buying rather than building permanently sacrifices core technical competencies
“When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like startups are so limited on time.”
Mohan: GPU container sharing limitations leave hardware heavily idle
“For most people, one of the things about CPUs that's really nice is with containers, right? You can end up having a single node and you can place many containers on them and all the containers will slowly start eating the compute. It's not really the same with…”
Mohan: Enterprises should fine-tune off-the-shelf models over custom architectures
“For a vast majority of enterprises, they should probably be using something off the shelf, fine tuning BERT models. If it's a vision, they should be fine tuning resonant or using something like clip, like the less work they can do the better.”
Mohan: Few products match Copilot due to compute-intensive low latency
“There's like a gap there where the number of products that have hit that co-pilot spot is actually not that high. A lot of them are kind of like wait and, you know, just generate a lot of stuff and see what happens because one is clearly more compute intensive…”
Mohan: Codeium scraped public code because The Pile was too small
“We ended up basically using a lot of open, I guess, permissively licensed code in the public internet, mainly because I think also the pile is, is fairly a small subset. I think maybe after we started, there was the stock that was also came to be, but for us, …”
Mohan: Codeium reached 10k users and 5% daily growth in late 2022
“So right now we have, we probably have over 10,000 users and thousands of daily actives and people come back day over day. Our growth is like around, you know, four to five percent day over day right now.”
Mohan: Automated PR generation will require specialized models trained on diffs
“A lot of things people are excited about right now are I write a comment and it generates a PR for me. And that's like really awesome in theory. I think that's like a really cool thing. And I'm sure at some point we will be able to get there. That will probabl…”
Mohan: Generating subsequent code blocks increases cognitive load during acceleration mode
“And a good heuristic is let's say you're inside like a basic block of a piece of code. Let's say you're inside a block of code or an if statement. You're probably already in acceleration mode and you would feel really bad if I started generating the else claus…”
Mohan: Marginal loss improvements unlock massive gains in model reasoning
“Like, small wins at the margins are massive wins in terms of IQ. Like, it's harder to get those, and they don't look as big, but they're, like, massive wins in terms of reasoning. They can now do chain of thought, all these other things.”
Mohan: Codeium to build embedding database infrastructure in-house
“I will just say for us, we will probably just build this in house to start with. And that's because I think the technology probably isn't there yet. And I think if the technology isn't there yet, like waiting on point solutions to come up is a lot harder than …”
Mohan: Evaluating coding models requires hundreds of thousands of completions
“It's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.”
Mohan: The Windsurf Editor is a direct fork of VS Code
“It is a fork of VS Code, and yes, We do know the memes. But there wasn't a need to rebuild everything from scratch.”
Mohan: JetBrains has by far the best debugger for Java
“Very honestly, JetBrains has the best debugger for Java. It's not even close, right?”
Mohan: Coding is iterative exploration, not converting PRDs into output
“Code is not really built as you have a PRD and then you get some output out. It's more like you have a general vision and yes. And as you write the code, you get more and more clarity on approaches that don't work and do work. You're killing ideas and creating…”
Mohan: Windsurf Cascade splits planning to Claude and codebase application internally
“The high level planning that is going on in the model is actually getting done with products like the cloud, but the extremely fast retrieval, as well as the ability to like take the high level plan and actually apply it to the code base is proprietary systems…”