Codeium

“We have over 800,000 sort of developers that use that product.”

“No, so actually the way it works is the high-level planning that is going on in the model is actually getting done with products like the Cloud. But the extremely fast retrieval, as well as the ability to, like, take the high-level plan and actually apply it t…”

“For a lot of the systems, we do believe embeddings work, but for complex questions, we don't believe embeddings can encapsulate all the granularity of a particular query. Like imagine, imagine I have a question on a codebase of find me all quadratic time algor…”

“We, by the way, we will never launch with a waitlist. We will never launch with a waitlist. That's the thing at the company. We'd much rather be a company that's considered the boring company than a company that, that launches once in a while, and like, hopefu…”

“Like I don't think we go from zero to ten million ARR in less than a year in an enterprise product if we don't have a product that developers love.”

“We sold the product to like 30 or 40 customers of that.”

“If we use VLLM, we would not be talking with you right now.”

“We believe that developers will never completely pose the problem statement, right? Because the problem statement lives in their head.”

“If we're actually being able to understand the kind of the trajectory of what developers are doing within the editor, right, if we actually are being able to see, like, oh, the developer just went and opened up this part of the directory and tried to view it, …”

“A lot of the benchmarks for these embedding-based systems are like needle in the haystack problems. Like, I want to find this one particular piece of information out of all this potential context. That's not That's not really what actually is necessary for doi…”

“We don't really like pay other providers to do things for indexing. We don't pay like other providers to do our serving of our own custom models, right? And I think that's a core competency within our company that we have decided to build, but that's also enab…”

“It previously was being leased by, I think, Facebook slash WhatsApp. And then immediately after that Ghost Autonomy, and then now here we are, and we also, you know, I guess one of the things that the landlord told us was this was the place that they shot all …”

“I actually think asking people to do things explicitly is probably going to be more of an anti-pattern if we can actually go and passively suggest the entire change for the user. So almost imagine as the user is using the product that we're going to suggest th…”

“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”

“If the agent there has to always get human involvement to keep on fixing itself, it's probably not ready to become a full end-to-end automated system, because then we're just going to turn into a linter where, like, it produces a bunch of things and no one loo…”

“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”

“The super complete product that we have has gone materially better because of us not only using synthetic data, but also getting the preference data from our users of, like, hey, given these set of trajectories, Here's actually what a good outcome is.”

“Once it starts hitting the peak of these benchmarks, getting that last 10% actually probably is like counterintuitive to the actual goal of what the benchmark was. Like you probably should find a new hill to climb rather than sort of p-hacking or really optimi…”

“If you want to run all of them in parallel, you probably don't want any machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writing ou…”

“The things like autocomplete and super complete that run on every keystroke are entirely like our own models. And by the way, that is still because properties like FIM fill in the middle capabilities are still quite bad with the current non-existent. They're a…”

“On VS Code, actually, the problem for us wasn't actually being able to implement the feature. We had the feature for a while. Problem was actually even to show the feature, VS Code would not expose an API for us to do this. So what we actually ended up doing w…”

“Like, we're still the only code assistant that has an extension of Eclipse. That's still true years in, right?”

“It's that idea that our computation is one 100th of the cost of the competitors. We are not using APIs, and as a result, our customers and our users actually get 100 X the amount of compute that they would on another product.”

“We ended up basically using a lot of open, I guess, permissively licensed code in the public internet, mainly because I think also the pile is, is fairly a small subset. I think maybe after we started, there was the stock that was also came to be, but for us, …”

“If we use VLLM, we would not be talking with you right now.”

“We threw some of the currently publicly available models at this notion of retrieval, this idea of using commit messages, and we found that there is reduced performance. They're unable to reason over specifically code, but then also specifically this kind of r…”

“When we're serving these products in an environment that works for these large enterprises, we're not going out and being like, we need to build this custom system for you guys, right? This is the same system that serves our entire user base.”

“So it's probably going to be a while before we even consider leaving code. Like that's going to be a big step because the amount of learning we need to do is going to be high. We need to get users, right? We've learned so much from our users already. So yeah, …”

“Yeah, we had sold the product to like, 30 or 40 customers of that kind.”

“We, by the way, we will never launch with a waitlist. We will never launch with a waitlist.”

“I will just say for us, we will probably just build this in house to start with. And that's because I think the technology probably isn't there yet. And I think if the technology isn't there yet, like waiting on point solutions to come up is a lot harder than …”

“And for our case, our models that we train internally, they might not be the most compute optimal. In other words, we probably could have had a better model by making it larger, but the trade-off would have been latency. We know what the impact of having highe…”

“The super complete product that we have has gone materially better because of us not only using synthetic data, but also getting the preference data from our users of, like, hey, given these set of trajectories, Here's actually what a good outcome is.”

“So we garnered a couple million users on our VS Code, JetBrains, Vim, Emacs extensions.”

“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight yea…”

“And we have heuristics in terms of capturing Whether or not you're in acceleration versus exploration mode.”

“MQuery runs thousands of LLMs in parallel, so the user can start streaming in code within seconds, not minutes, not hours, Seconds and oftentimes milliseconds.”

“And I guess maybe one of the things I'm a little bearish on is If another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing? What are they really doing at that point? If t…”

“But in the future, they're going to generate documentation. They're going to generate commit messages, code reviews, code scanning. They're going to take, you know, Figma artboards and convert them into component into UIs that were built by your own components…”

“Early on, we were able to leverage open source models, sort of bootstrap it internally within our company. But then to ship, we finally had some requirements like, Hey, we want this model to have fill in the middle capabilities and a bunch of other things. And…”

“Right now, you know, we're, have a lot of customers in, like, the defense space, for example. We're going through, you know, FedRAMP accreditation right now”

“The product is actually one of those products where even use Copilot and use us, it's hard to tell the difference actually. And a lot of our users have actually churned off of Copilot.”

“We don't really like pay other providers to do things for our indexing. We don't pay like other providers to do our serving of our own customer models, right?”

“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”

“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”

“So in three months, we've been generating 4.5 billion lines of code.”

“And if you want to run all of them in parallel, you probably don't want end machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writin…”

“The things like autocomplete and supercomplete that run on every keystroke are entirely, like, our own models, and by the way, that is still because properties like FIM, fill in the middle capabilities are still quite bad with the current model.”

“I don't think we go from zero to ten million ARR in less than a year in an enterprise product if we don't have a product that developers love.”

“It's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.”

“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”

“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”

“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”

“We are the highest rated developer tool as voted in by developers in the most recent stack overflow survey. And you'll note that this is even higher than tools like chat GPT and GitHub copilot.”

“We have over 800,000 sort of developers that use that product.”

“We've been downloaded over a million and a half times. We're one of the top rated extensions across the different marketplaces. And to reiterate, reiterate, we offer free unlimited autocomplete chat and search across 70 different languages and 40 different IDE…”

“So right now we have, we probably have over 10,000 users and thousands of daily actives and people come back day over day. Our growth is like around, you know, four to five percent day over day right now.”

“It's evident through the years that we're actually hitting a ceiling on what is possible using these traditional vector embeddings. And over time, even the biggest models are approximating to around the same level of performance.”

“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality…”

“Problem was actually even to show the feature VS code would not expose an API for us to do this. So what we actually ended up doing was dynamically generating PNGs to actually go out and showcase this.”

“And so we just created a 10 dollar a month pro plan, mostly just to cover the cost. Like we're not really like operating, I think, on a much of a margin there either, but like, okay, like just to cover us there.”