Codeium

“We ended up basically using a lot of open, I guess, permissively licensed code in the public internet, mainly because I think also the pile is, is fairly a small subset. I think maybe after we started, there was the stock that was also came to be, but for us, …”

“Early on, we were able to leverage open source models, sort of bootstrap it internally within our company. But then to ship, we finally had some requirements like, Hey, we want this model to have fill in the middle capabilities and a bunch of other things. And…”

“So right now we have, we probably have over 10,000 users and thousands of daily actives and people come back day over day. Our growth is like around, you know, four to five percent day over day right now.”

“The product is actually one of those products where even use Copilot and use us, it's hard to tell the difference actually. And a lot of our users have actually churned off of Copilot.”

“And if I were to go out and think about like the three categories of like an LLM product, it's probably like latency, quality, and correctability. The reality is, you know, if I were to take a product like Copilot or Kodia, the latency is very low. The quality…”

“And we have heuristics in terms of capturing Whether or not you're in acceleration versus exploration mode.”

“So it's probably going to be a while before we even consider leaving code. Like that's going to be a big step because the amount of learning we need to do is going to be high. We need to get users, right? We've learned so much from our users already. So yeah, …”

“And for our case, our models that we train internally, they might not be the most compute optimal. In other words, we probably could have had a better model by making it larger, but the trade-off would have been latency. We know what the impact of having highe…”

“I will just say for us, we will probably just build this in house to start with. And that's because I think the technology probably isn't there yet. And I think if the technology isn't there yet, like waiting on point solutions to come up is a lot harder than …”

“One of the issues that ends up coming up with things like human eval is contamination, because a lot of these things that train models end up training on all of GitHub. GitHub itself has human eval. So they end up Training on that, and then the numbers are arb…”

“And I guess maybe one of the things I'm a little bearish on is If another company comes out and solves eval properly for a bunch of different verticals, what was the company that they were selling to really doing? What are they really doing at that point? If t…”

“It's honestly really hard to get signal unless you have like a non-trivial amount of usage. Non-trivial meaning you're getting, you're doing hundreds of thousands of completions, if not millions of completions.”

“We've been downloaded over a million and a half times. We're one of the top rated extensions across the different marketplaces. And to reiterate, reiterate, we offer free unlimited autocomplete chat and search across 70 different languages and 40 different IDE…”

“We are the highest rated developer tool as voted in by developers in the most recent stack overflow survey. And you'll note that this is even higher than tools like chat GPT and GitHub copilot.”

“It's evident through the years that we're actually hitting a ceiling on what is possible using these traditional vector embeddings. And over time, even the biggest models are approximating to around the same level of performance.”

“We threw some of the currently publicly available models at this notion of retrieval, this idea of using commit messages, and we found that there is reduced performance. They're unable to reason over specifically code, but then also specifically this kind of r…”

“It's that idea that our computation is one 100th of the cost of the competitors. We are not using APIs, and as a result, our customers and our users actually get 100 X the amount of compute that they would on another product.”

“MQuery runs thousands of LLMs in parallel, so the user can start streaming in code within seconds, not minutes, not hours, Seconds and oftentimes milliseconds.”

“But in the future, they're going to generate documentation. They're going to generate commit messages, code reviews, code scanning. They're going to take, you know, Figma artboards and convert them into component into UIs that were built by your own components…”

“We have over 800,000 sort of developers that use that product.”

“Problem was actually even to show the feature VS code would not expose an API for us to do this. So what we actually ended up doing was dynamically generating PNGs to actually go out and showcase this.”

“And if you do this across enough repositories, you've turned this highly discontinuous and discrete problem of make a PR work versus make it not work into a continuous problem. And now that's a hill you can actually climb.”

“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”

“The things like autocomplete and supercomplete that run on every keystroke are entirely, like, our own models, and by the way, that is still because properties like FIM, fill in the middle capabilities are still quite bad with the current model.”

“We don't really like pay other providers to do things for our indexing. We don't pay like other providers to do our serving of our own customer models, right?”

“When we're serving these products in an environment that works for these large enterprises, we're not going out and being like, we need to build this custom system for you guys, right? This is the same system that serves our entire user base.”

“I think that right now optimizing for making money off of individual developers is probably the wrong, actually, strategy. Largely because I think individual developers can switch off of products, like, very quickly, and unless we have, like, a very large lead…”

“And so we just created a 10 dollar a month pro plan, mostly just to cover the cost. Like we're not really like operating, I think, on a much of a margin there either, but like, okay, like just to cover us there.”

“And if you want to run all of them in parallel, you probably don't want end machines to go out and do it. I think that's unnecessary. Especially if most of them are I.O. Bound kind of operations where all you're doing is reading a little bit of data and writin…”

“I don't think we go from zero to ten million ARR in less than a year in an enterprise product if we don't have a product that developers love.”

“I don't think most of us got much value from Chachapi team. Largely because I think the problem was, and this is maybe a little bit of a different thing, it's, like, a lot of the engineers at the company who have been writing software for, like, over eight yea…”

“We, by the way, we will never launch with a waitlist. We will never launch with a waitlist.”

“The super complete product that we have has gone materially better because of us not only using synthetic data, but also getting the preference data from our users of, like, hey, given these set of trajectories, Here's actually what a good outcome is.”

“One of the really beautiful parts about our product that is very different than a chat GPT is we can not only see if the acceptance happened, but if something more than the acceptance happened and it happened even more than that, right? Like, let's say you acc…”

“Right now, you know, we're, have a lot of customers in, like, the defense space, for example. We're going through, you know, FedRAMP accreditation right now”

“If we use VLLM, we would not be talking with you right now.”

“Yeah, we had sold the product to like, 30 or 40 customers of that kind.”

“So we garnered a couple million users on our VS Code, JetBrains, Vim, Emacs extensions.”

“So in three months, we've been generating 4.5 billion lines of code.”

“Every year they induct, I think one or two companies into their hall of innovation. So these are like companies that have made the sort of biggest impact on JP Morgan. And Codium was one of those companies.”

“And in the middle of two, two rolled around and we had a couple million in revenue that we were managing upwards of 10,000 sort of GPUs. We had eight people at the time we were free cashflow positive.”

“So VS Code, which is a very popular IDE, had a ceiling for the AI capabilities we could showcase our users. And because of that, we decided to go out and fork VS Code and build our own IDE with some of these new agentic capabilities”

“We had already raised twenty eight million dollars of capital. And we were just like, Hey, we're just going to pivot entirely from this. And we did that overnight, right?”

“This year, the total amount of engineering output we will have is much larger than the engineering output we've had since the beginning of the company's creation till now.”

“So we have close to a 160 people on the engineering team is over 50 people.”

“And I think one of the biggest dog whistles I sort of hear is when I ask people like, how hard are you willing to work? Some people actually ultimately say, Hey, I work very smart. And I basically asked them a question. If we have many smart people at our comp…”

“The thing about a person that is, is not pulling their weight, that's bad is not, it's not the productivity, right? At some point when the company becomes many hundreds of engineers, I'm not going to be thinking about the one engineer that's not pulling their …”

“Yeah, it's probably post or take home. It's probably that actually. So the take home itself filters probably another, you know, 10, 15 X on top of that.”

“We're okay with people using the tools because I think one of the worst things is like, if someone comes here and doesn't like using these tools, like we believe they're massive productivity improvements.”

“And the go-to-market team is now over 80 people inside the company.”

“We have custom models that we built that are post trained on top of popular open source models that can make these edits really, really quickly to the code base. And the reason why you would want to do that is it's a faster and be also that model can actually …”

“Interestingly, the one that does retrieval is actually completely pre-trained in house. That actually does that. But yeah, for a lot of different pieces, it's based on open source for interestingly, for the one that does that does the edits and autocomplete th…”

“It's that actually every hour we get probably tens of millions of pieces of feedback from our users. We get a lot of feedback on what they like and what they don't like.”

“One interesting decision that we kind of have for core engineering is that we don't have pure product managers for the core engineering side of the company. And by the way, that's purely because we build for developers and our product is, is built by developer…”

“One of the goals that I tell everyone at our company is we should be cannibalizing the existing state of our product every six to 12 months. Every six to 12 months, it should make our existing product look silly. It should almost make the form factor of existi…”

“We were the first identic IDE product out there, right? That's what we landed with.”

“Our first product, which is basically a plugin that works across any IDE, 70 plus languages on average generates 45% of the code written by the developers using it.”

“Our founders did a couple million dollars with founder led sales before hiring anyone.”

“One of the things that we're doing going into next year, based on all the data that we've seen this year, is moving to a partner first model for revenue.”

“Like for any, I'd say for any founder listening to this, like you hire a sales leader that doesn't have four to six people that can basically text and get into the seat and like, Four to six weeks, probably even sooner, let's say four weeks, then like maybe no…”