The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 61 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Mohan: Bearish on outsourced eval startups because AI companies must own evaluations
“And I guess maybe one of the things I'm a little bearish on is If another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing? What are they really doing at that point? If t…”
Ramachandran: SWE-bench and HumanEval do not reflect real professional software engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”
Varun Mohan: Codeium would have failed if it used vLLM
“If we use VLLM, we would not be talking with you right now.”
Ramachandran: SWE-bench and HumanEval do not reflect real professional engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”
Hou: Codeium inference costs 1/100th of competitors by avoiding third-party APIs
“It's that idea that our computation is one 100th of the cost of the competitors. We are not using APIs, and as a result, our customers and our users actually get 100 X the amount of compute that they would on another product.”
Mohan: Codeium quality matches Copilot and drives user churn
“The product is actually one of those products where even use Copilot and use us, it's hard to tell the difference actually. And a lot of our users have actually churned off of Copilot.”
Mohan: Monetizing individual developers in AI tooling is the wrong strategy
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Mohan: Codeium uses in-house models for autocomplete due to poor frontier FIM
“The things like autocomplete and supercomplete that run on every keystroke are entirely, like, our own models, and by the way, that is still because properties like FIM, fill in the middle capabilities are still quite bad with the current model.”
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Mohan: Codeium generated dynamic PNGs due to VS Code API limitations
“On VS Code, actually, the problem for us wasn't actually being able to implement the feature. We had the feature for a while. Problem was actually even to show the feature, VS Code would not expose an API for us to do this. So what we actually ended up doing w…”
Mohan: Codeium would not have survived if it relied on vLLM
“If we use VLLM, we would not be talking with you right now.”
Mohan: Codeium trains smaller, compute-suboptimal models for lower inference latency
“And for our case, our models that we train internally, they might not be the most compute optimal. In other words, we probably could have had a better model by making it larger, but the trade-off would have been latency. We know what the impact of having highe…”
Mohan: Senior engineers gained little value from ChatGPT due to context friction
“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight yea…”
Mohan: Stripping open-source commits turns coding agent evals into a continuous problem
“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”
Kevin Hou: Traditional vector embeddings have hit a performance ceiling
“It's evident through the years that we're actually hitting a ceiling on what is possible using these traditional vector embeddings. And over time, even the biggest models are approximating to around the same level of performance.”
Mohan: LLM products depend on latency, quality, and correctability
“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality…”
Mohan: Codeium bypassed VS Code API limits by rendering Supercomplete via PNGs
“Problem was actually even to show the feature VS code would not expose an API for us to do this. So what we actually ended up doing was dynamically generating PNGs to actually go out and showcase this.”
Mohan: Windsurf uses Claude for planning, proprietary models for retrieval and diffs
“No, so actually the way it works is the high-level planning that is going on in the model is actually getting done with products like the Cloud. But the extremely fast retrieval, as well as the ability to, like, take the high-level plan and actually apply it t…”
Mohan: Embeddings fail at complex codebase retrieval tasks like identifying quadratic algorithms
“For a lot of the systems, we do believe embeddings work, but for complex questions, we don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a codebase of find me all quadratic time algor…”
Mohan: Developers will never provide complete problem statements to AI
“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”
Ramachandran: Tracking developer trajectory allows AI to infer intent without prompting
“If we're actually being able to understand the kind of the trajectory of what developers are doing within the editor, right, if we actually are being able to see, like, oh, the developer just went and opened up this part of the directory and tried to view it, …”
Mohan: Codeium builds proprietary indexing and serving rather than paying third parties
“We don't really like pay other providers to do things for indexing. We don't pay like other providers to do our serving of our own custom models, right? And I think that's a core competency within our company that we have decided to build, but that's also enab…”
Mohan: Explicit user prompting will become an AI coding anti-pattern
“I actually think asking people to do things explicitly is probably going to be more of an anti-pattern if we can actually go and passively suggest the entire change for the user. So almost imagine as the user is using the product that we're going to suggest th…”
Mohan: Squeezing the final 10% on AI benchmarks encourages p-hacking
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is like counterintuitive to the actual goal of what the benchmark was. Like you probably should find a new hill to climb rather than sort of p-hacking or really optimi…”
Mohan: Parallel multi-agent coding trajectories do not require separate remote machines
“If you want to run all of them in parallel, you probably don't want any machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writing ou…”
Mohan: Codeium scraped public code because The Pile was too small
“We ended up basically using a lot of open, I guess, permissively licensed code in the public internet, mainly because I think also the pile is, is fairly a small subset. I think maybe after we started, there was the stock that was also came to be, but for us, …”
Hou: Public retrieval models underperform on git commit benchmarks
“We threw some of the currently publicly available models at this notion of retrieval, this idea of using commit messages, and we found that there is reduced performance. They're unable to reason over specifically code, but then also specifically this kind of r…”
Varun Mohan: Codeium serves individual users and large enterprises with the exact same system
“When we're serving these products in an environment that works for these large enterprises, we're not going out and being like, we need to build this custom system for you guys, right? This is the same system that serves our entire user base.”
Mohan: Codeium will never launch a product with a waitlist
“We, by the way, we will never launch with a waitlist. We will never launch with a waitlist.”
Mohan: Codeium to build embedding database infrastructure in-house
“I will just say for us, we will probably just build this in house to start with. And that's because I think the technology probably isn't there yet. And I think if the technology isn't there yet, like waiting on point solutions to come up is a lot harder than …”
Hou: Codeium's mQuery runs thousands of LLMs in parallel for millisecond latency
“MQuery runs thousands of LLMs in parallel, so the user can start streaming in code within seconds, not minutes, not hours, Seconds and oftentimes milliseconds.”
Hou: Future AI tools will generate docs, reviews, and UIs from Figma
“But in the future, they're going to generate documentation. They're going to generate commit messages, code reviews, code scanning. They're going to take, you know, Figma artboards and convert them into component into UIs that were built by your own components…”
Varun Mohan: Codeium does not pay third parties for indexing or custom model serving
“We don't really like pay other providers to do things for our indexing. We don't pay like other providers to do our serving of our own customer models, right?”
Hou: Windsurf generated 4.5 billion lines of code in three months
“So in three months, we've been generating 4.5 billion lines of code.”
Varun Mohan: Parallel multi-agent coding trajectories don't require multiple machines
“And if you want to run all of them in parallel, you probably don't want end machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writin…”
Ramachandran: Codeium grew from zero to $10M ARR in under a year
“I don't think we go from zero to ten million ARR in less than a year in an enterprise product if we don't have a product that developers love.”
Mohan: Evaluating coding models requires hundreds of thousands of completions
“It's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.”
Mohan: Post-acceptance code edits provide better training signal than mere acceptance rates
“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”
Hou: Codeium is highest-rated dev tool in Stack Overflow survey
“We are the highest rated developer tool as voted in by developers in the most recent stack overflow survey. And you'll note that this is even higher than tools like chat GPT and GitHub copilot.”
Mohan: Codeium reached 10k users and 5% daily growth in late 2022
“So right now we have, we probably have over 10,000 users and thousands of daily actives and people come back day over day. Our growth is like around, you know, four to five percent day over day right now.”
Mohan: Codeium will never launch a product with a waitlist
“We, by the way, we will never launch with a waitlist. We will never launch with a waitlist. That's the thing at the company. We'd much rather be a company that's considered the boring company than a company that, that launches once in a while, and like, hopefu…”
Ramachandran: Codeium reached $10M enterprise ARR in under a year
“Like I don't think we go from zero to ten million ARR in less than a year in an enterprise product if we don't have a product that developers love.”
Ramachandran: Needle-in-a-haystack retrieval benchmarks fail to model real code retrieval
“A lot of the benchmarks for these embedding-based systems are like needle in the haystack problems. Like, I want to find this one particular piece of information out of all this potential context. That's not That's not really what actually is necessary for doi…”
Ramachandran: Premature end-to-end AI automation reduces agents to ignored linters
“If the agent there has to always get human involvement to keep on fixing itself, it's probably not ready to become a full end-to-end automated system, because then we're just going to turn into a linter where, like, it produces a bunch of things and no one loo…”
Mohan: IDEs evaluate AI models better by tracking post-acceptance edits
“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”
Mohan: Codeium uses proprietary autocomplete models due to poor frontier LLM FIM
“The things like autocomplete and super complete that run on every keystroke are entirely like our own models. And by the way, that is still because properties like FIM fill in the middle capabilities are still quite bad with the current non-existent. They're a…”
Ramachandran: Codeium is the only AI code assistant supporting Eclipse
“Like, we're still the only code assistant that has an extension of Eclipse. That's still true years in, right?”
Mohan: Codeium will not expand outside code into legal AI anytime soon
“So it's probably going to be a while before we even consider leaving code. Like that's going to be a big step because the amount of learning we need to do is going to be high. We need to get users, right? We've learned so much from our users already. So yeah, …”