The Exchanges

Q You have made a huge bet on AI as a company. Can you just talk about Ghostwriter and how it came about and the investment in this area?

A Yeah, throughout my career working on code that handles code, right? Whether it's at Codecademy for teaching programming, whether it's my own project, whether it's React and building the runtime around React Native, I always felt like our tools that were handling code, whether it's like compiling it, parsing it, minifying it, all that stuff. We're very kind of rigid and very laborious. Um, in a lot of ways you're building sort of a classical intelligence system, very algorithmic, um, and a lot of heuristics and, and all of that. And I always thought that you can probably apply machine learning to it. And I started reading around whether anyone had done it. There was this seminal paper in 2012 called on the naturalness of software. And basically a bunch of researchers try to apply NLP to code. And what they found is that actually code Can be modeled like any language. That's why they call it naturalness is because hey, code is kind of repetitive, like language. You can do things like Ingram, basic Ingram model can actually start to generate code that is compilable. That had a huge impact on me. And every year or two while starting Replit, I would go look at the state of the art on ML on code and nothing ever really worked all that well up until GPT-II. And you can like take GPT-II and fine tune it and like try to make it write code and was like kind of okay. But obviously GPT-II…

AI assessment note: “And that's when we started building what became Ghostwriter.”

Q code autocomplete with local context has become as widely adopted as it has. Like your tool and co-pilot are sort of a handful of tools that people like very much accept have changed workflow. Maybe if we just project out a little bit, where do you think that the like next big leaps in productivity in software development come from in terms of AI? What else is going to happen?

A It's an amazing tool, but I think it's still very primitive, and we haven't fully explored the full capabilities of even GPT, but also transformer models applied to code in general. There's a lot to do. I think there's a lot to build on the sort of the layer just around the models. So they're basically how to give models better context, how to give them tools, the ability for the model to actually Be able to go out and read a file, install a package, evaluate code, the ability for these models to be more agentic and be able to kind of write entire features, uh, by themselves. And so that that's all the work just around the model itself with the model in the center. And then there's a lot of work on the models themselves that haven't been, you know, fully explored. The way we train models today is we just give it a large corpus of, of, of code. But you can imagine different ways of training models, and we've played around with different techniques at Replit. There are ways to make the models, give models more intuition about how code execution happens as opposed to just static code, because you're training them on static code. They understand the structure and syntax of code, but they don't really understand the semantics of code. So you can imagine a way to train Models where you're not only feeding them code, but you're also feeding them the results of the evaluation of that c…

AI assessment note: “how to give models better context, how to give them tools”

Q Can you talk about the decision to, um, train models versus use existing APIs? Like, was it just a latency thing? How did that happen?

A So when Microsoft did it, like, they took GPT-III, they distilled it down, and they hosted it on their own infrastructure. They did a bunch of, uh, caching, a lot of things like that. If you're someone who's just using the OpenAI API, you couldn't do any of that. And it was very expensive. It was very slow. Um, and even now, you know, there's no completion models. Now all the models are chat models. You know, they're not going to be releasing any completion model going forward. And so if you want to build a completion based product, it's actually fairly difficult to do it using commercial APIs. And we felt like we can, we can make a model that's both cheap and fast and good enough for that autocomplete use case. And we found that the three billion parameter size is kind of the nice sweet spot where you get enough raw IQ from it, because like one billion felt kind of too dumb in a lot of ways. And it was going to be cheap enough and fast enough to host and be able to do a fast inference from. Um, and it was a time after Yeah, the chinchilla paper first came out and LLAMA had been announced and the idea of just training them longer had just been in the air and, and we're like, okay, what if we, we apply all these open source tokens on a smaller model? And then we applied some more tokens from replets data that gave us a 50% improvement over the model that we open source. So we op…

AI assessment note: “we felt like we can, we can make a model that's both cheap and fast”

Q for training your own models, for reinforcement learning, et cetera. There's a paper that recently came out from Google comparing a very specific type of reinforcement learning through human feedback versus AI feedback, and it roughly netted out, right? You could start to use AI for certain tasks to Effectively train itself. Like, how do you think about that in the context of code and sort of where you're heading?

A I think it's easy to overstate data modes. You know, you need to be measured about that. I think the real advantage of Replit is actually just the platform, just the end to end platform. Because I think, you know, you can be, you can try to build these code models, but without having the platform to be able to apply it on, to be able to get feedback from, it gets very difficult. But also, even if you just have a Code editor or editor extension. That doesn't give you all the feedback that you would want, because you would want feedback from deployment. You would want feedback from execution of the, of the code, and not just user feedback. I think user feedback, you're right for RLHF type things. I don't think that's the real advantage. The real advantage is the end-to-end journey from the first line of code to the deployment, to getting crashes in production, to making edits, to all of that feedback cycle, which typically in most cases is divided between GitHub and VS Code and AWS and all of that. And the real magic of Replet is putting all of that together. And this is, I think, where we're going to get the kind of Richness of training data that allows you to train over a large action space, and that's, that's really exciting.

AI assessment note: “The real advantage is the end-to-end journey from the first line of code to the deployment”

Q Because you're somebody who like clearly believes in agentic development, and then you stuck a marketplace in your product that allows developers to collect dollars for code that they've produced. Do you think we end up doing that with development agents or how does that play out?

A So, uh, the idea behind bounties is that at Revlet, we have all this talented people across the world. That are learning how to code and that are actually like becoming great programmers that kind of no one knows about. And I felt like it was almost our duty to, you know, make it known that there's so much talent around the world that you can kind of leverage. And a lot of these people don't have any resumes, don't have a lot of skills to be able to get, go get jobs themselves. So part of it was, there's a, Community building activity. Can we get all these people money? And like we have people in Africa, uh, India, Thailand, uh, Serbia, all sorts of countries now that are like making money on bounties. The other thing was my view at the time or my thesis that I wanted to test was it's going to be a lot easier and it's going to be a lot cheaper to be able to get basic software tasks done because AI, like I said, will have in the short term, the most impact on beginners. So beginners are actually getting a two, three, 10 X multiplication on their productivity. You learn a little bit of code and now you can actually build a product that makes a hundred K RR. And so the idea was that, okay, how do we leverage this, uh, this opportunity? How do we monetize it? And so, okay, human plus AI, the human who knows a little bit of code plus AI could actually be a really great And so yeah, …

AI assessment note: “okay, human plus AI, the human who knows a little bit of code plus AI”

Q in terms of catching up relative to Uh, GPT-FOR and sort of other closed source models. What do you think happens if, if Facebook stops open sourcing? Do you think somebody else steps into the void? Do you think everybody's kind of on their own? Like, I'm a little bit curious how you think about the open source world relative to models given how important LAMA has become so rapidly.

A So when I came to the realization early 22 that we can't build the product That we want to build using commercial APIs. I was kind of depressed for a moment. It was like, oh, you know, we didn't have, you know, a lot of funding or whatever is like, okay, you know, Microsoft's going to be the only ones going to be able to build these kinds of products. And, uh, I got a meeting with Zuck actually, and I pitched Zuck on this idea. And I remember around the time he did the open compute project. So what open compute was like Google and AWS are going to keep all their data center secrets because it's a, it's a company advantage. But Facebook did not sell compute, did not sell cloud. And there's a lot of other companies that did not sell cloud. Cloud was a compliment for them, was not a competitive advantage. And so they pooled their resources together and they worked on this open compute project to be able to optimize computers for data centers. And it was a huge success. And what I told Zuck at the time was like, hey, why didn't you do that for LLMs? Obviously, Open source AI is going to be great for you because you can build better product. It's a compliment. Uh, and so it's sort of the traditional, like the classical, like commoditize your compliment, a business strategy. And at the time he's just, uh, kind of nodded his head and I don't know if it was a novel idea, maybe he was a…

AI assessment note: “So I think it makes a lot of sense for them to do that.”