The Wisdom Wall

“the open source serving. Offerings are just, I will say not great in that, in that they aren't customized to transformers and these kinds of workloads where I have high latency and I want to like batch requests and I want to batch requests while keeping latency low. But one of the weird things about generation models…”

“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arbitrarily a lot higher.”

“once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is, like, counterintuitive to the actual goal of what the benchmark was. Like, you probably should find a new hill to climb, rather than sort of p-hacking or really optimizing for how you can get higher on the benchmark.”

“For a lot of the systems, we do believe embeddings work, but for complex questions, We don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a code base of find me all quadratic time algorithms in this code base. Do we genuinely believe the…”

“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead trying to optimize for making a lot of profit off of…”

“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead trying to optimize for making a lot of profit off of…”

“Once you start using products like this, where in the beginning there's like, there's like skepticism, like how, how valuable can it be? And suddenly now like user behavior fundamentally changes so that now when I need to write a function, I'm like documenting my code more because I think it's prompting the model…”

“unless you are, you probably should always bootstrap on top of an existing API, right? Because like, even if you were to, the only reason why we didn't is because we knew that this product was actually buildable. Probably if we worked hard enough to train a model, we would actually be able to build a great product…”

“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality I think is good enough for the task, but the correctability…”

“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”

“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”

“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight years, And this is not to say they know everything that ChaiGPT…”

“When you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like, startups are so limited on time.”

“GitHub has very high switching costs or source code management tools, right? Because you actually need to switch over all the dependent systems on this workflow software. It's much harder than even switching off of a database.”

“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”

“once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is like counterintuitive to the actual goal of what the benchmark was. Like you probably should find a new hill to climb rather than sort of p-hacking or really optimizing for how you can get higher on the benchmark.”

“For a lot of the systems, we do believe embeddings work, but for complex questions, we don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a codebase of find me all quadratic time algorithms in this codebase. Do we genuinely believe the…”

“if you want to run all of them in parallel, you probably don't want any machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. bound kind of operations where all you're doing is reading a little bit of data and writing out a little bit of data. It's not extremely compute…”

“when you go out and buy, you're losing a core competency inside the company. And that's a core competency you can never get. It's, or it's very hard. Like startups are so limited on time.”

“for most people, one of the things about CPUs that's really nice is with containers, right? You can end up having a single node and you can place many containers on them and all the containers will slowly start eating the compute. It's not really the same with GPUs. Like let's say you have a single node. For the most…”

“for a vast majority of enterprises, they should probably be using something off the shelf, fine tuning BERT models. If it's a vision, they should be fine tuning resonant or using something like clip, like the less work they can do the better.”

“there's like a gap there where the number of products that have hit that co-pilot spot is actually not that high. A lot of them are, are kind of like wait and, you know, just generate a lot of stuff and see what happens because one is clearly more compute intensive than the other basically.”

“And a good heuristic is let's say you're inside like a basic block of a piece of code. Let's say you're inside a block of code or an if statement. You're probably already in acceleration mode and you would feel really bad if I started generating the else clause. Because what happens if that else clause is really wrong?…”

“Like, small wins at the margins are massive wins in terms of IQ. Like, it's harder to get those, and they don't look as big, but they're, like, massive wins in terms of reasoning. They can now do chain of thought, all these other things.”

“it's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.”

“Code is not really built as you have a PRD and then you get some, some output out. It's more like you have a general vision and yes. And as you write the code, you get more and more clarity on approaches that don't work and do work. You're killing ideas and creating ideas constantly.”

“Individual developers are much more sort of tuned towards small price changes. They care a lot more, right? Like if our product is 10, 20 bucks a month instead of 50 or a hundred bucks a month, that matters to them a lot. And for a large company where they're already spending billions of dollars on software, this is…”

“Microsoft Word is a crazy product. It made it so that the moment you review anything in Microsoft Word, the only way you can review it is with other people in Microsoft Word. It's like this virus that penetrates everything, and it's not only penetrates it within the company, it penetrates it across company too.”

“And if you want to run all of them in parallel, you probably don't want end machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. bound kind of operations where all you're doing is reading a little bit of data and writing out a little bit of data. It's not extremely compute…”

“one of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you accepted it, but then after accepting it, you deleted three or…”

“one of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you accepted it, but then after accepting it, you deleted three or…”

“when you have everyone sort of product oriented on the consumer side, the enterprise is just, it's checking off a lot of boxes that ultimately do not help the consumer at all. Doesn't help your growth metrics. And effectively, if the original group of people didn't care, it's incredibly hard to get them to care down…”