The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 32 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Mohan: Off-the-shelf serving frameworks leave significant FLOP utilization on the table
“The open source serving. Offerings are just, I will say not great in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. Bu…”
Mohan: HumanEval benchmark scores are inflated due to GitHub training contamination
“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”
Mohan: Squeezing the last 10% from AI benchmarks is counterproductive
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really op…”
Mohan: Vector embeddings fail at granular code retrieval queries
“For a lot of the systems, we do believe embeddings work, but for complex questions, We don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a code base of find me all quadratic time algo…”
Mohan: Monetizing individual developers in AI tooling is the wrong strategy
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Mohan: Monetizing individual developers is flawed due to low switching costs
“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”
Mohan: AI coding tools create a self-fulfilling loop by changing developer behavior
“Once you start using products like this, where in the beginning there's like skepticism, like how, how valuable can it be? And suddenly now like user behavior fundamentally changes so that now when I need to write a function, I'm like documenting my code more …”
Mohan: AI startups should bootstrap on existing APIs unless fine-tuning on large datasets
“Unless you are, you probably should always bootstrap on top of an existing API, right? Because like, even if you were to, the only reason why we didn't is because we knew that this product was actually buildable. Probably if we worked hard enough to train a mo…”
Mohan: LLM products depend on latency, quality, and correctability
“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality…”
Mohan: Stripping open-source commits turns coding agent evals into a continuous problem
“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”
Mohan: Developers will never completely articulate problem statements to AI
“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”
Mohan: Senior engineers gained little value from ChatGPT due to context friction
“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight yea…”
Mohan: Buying third-party software risks permanently losing core engineering competencies
“When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like, startups are so limited on time.”
Mohan: Switching SCM tools is harder than switching databases
“GitHub has very high switching costs or source code management tools, right? Because you actually need to switch over all the dependent systems on this workflow software. It's much harder than even switching off of a database.”
Mohan: Developers will never provide complete problem statements to AI
“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”
Mohan: Squeezing the final 10% on AI benchmarks encourages p-hacking
“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is like counterintuitive to the actual goal of what the benchmark was. Like you probably should find a new hill to climb rather than sort of p-hacking or really optimi…”
Mohan: Embeddings fail at complex codebase retrieval tasks like identifying quadratic algorithms
“For a lot of the systems, we do believe embeddings work, but for complex questions, we don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a codebase of find me all quadratic time algor…”
Mohan: Parallel multi-agent coding trajectories do not require separate remote machines
“If you want to run all of them in parallel, you probably don't want any machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writing ou…”
Varun Mohan: Buying rather than building permanently sacrifices core technical competencies
“When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like startups are so limited on time.”
Mohan: GPU container sharing limitations leave hardware heavily idle
“For most people, one of the things about CPUs that's really nice is with containers, right? You can end up having a single node and you can place many containers on them and all the containers will slowly start eating the compute. It's not really the same with…”
Mohan: Enterprises should fine-tune off-the-shelf models over custom architectures
“For a vast majority of enterprises, they should probably be using something off the shelf, fine tuning BERT models. If it's a vision, they should be fine tuning resonant or using something like clip, like the less work they can do the better.”
Mohan: Few products match Copilot due to compute-intensive low latency
“There's like a gap there where the number of products that have hit that co-pilot spot is actually not that high. A lot of them are kind of like wait and, you know, just generate a lot of stuff and see what happens because one is clearly more compute intensive…”
Mohan: Generating subsequent code blocks increases cognitive load during acceleration mode
“And a good heuristic is let's say you're inside like a basic block of a piece of code. Let's say you're inside a block of code or an if statement. You're probably already in acceleration mode and you would feel really bad if I started generating the else claus…”
Mohan: Marginal loss improvements unlock massive gains in model reasoning
“Like, small wins at the margins are massive wins in terms of IQ. Like, it's harder to get those, and they don't look as big, but they're, like, massive wins in terms of reasoning. They can now do chain of thought, all these other things.”
Mohan: Evaluating coding models requires hundreds of thousands of completions
“It's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.”
Mohan: Coding is iterative exploration, not converting PRDs into output
“Code is not really built as you have a PRD and then you get some output out. It's more like you have a general vision and yes. And as you write the code, you get more and more clarity on approaches that don't work and do work. You're killing ideas and creating…”
Varun Mohan: Individual devs are price-sensitive; enterprises tolerate higher software costs
“Individual developers are much more sort of tuned towards small price changes. They care a lot more, right? Like if our product is 10, 20 bucks a month instead of 50 or a hundred bucks a month, that matters to them a lot. And for a large company where they're …”
Mohan: Microsoft Word spreads virally across companies through forced review compatibility
“Microsoft Word is a crazy product. It made it so that the moment you review anything in Microsoft Word, the only way you can review it is with other people in Microsoft Word. It's like this virus that penetrates everything, and it's not only penetrates it with…”
Varun Mohan: Parallel multi-agent coding trajectories don't require multiple machines
“And if you want to run all of them in parallel, you probably don't want end machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writin…”
Mohan: Post-acceptance code edits provide better training signal than mere acceptance rates
“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”
Mohan: IDEs evaluate AI models better by tracking post-acceptance edits
“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”
Varun Mohan: Consumer-first companies struggle to retrofit enterprise DNA later
“When you have everyone sort of product oriented on the consumer side, the enterprise is just, it's checking off a lot of boxes that ultimately do not help the consumer at all. Doesn't help your growth metrics. And effectively, if the original group of people d…”