The Exchanges

Q Um, what's driving that? What are you finding?

A So, We have been rolling people off the waitlist that we sort of amassed when we announced when I last saw you. Um, and it's been really interesting because as I may have mentioned on the podcast, like we had to be very opinionated about the data mix and the data set that we put together for like sort of the V zero of Gini. Um, again, like to your point, JavaScript, JavaScript, JavaScript, Python, right? There's a lot of JavaScript and it's various forms in there. Um, But it turns out that when we've shipped it, um, to the, to the very early alpha users we rolled it out to, um, for example, we had some guys using it, um, with a C-sharp code base, and C-sharp currently represents, I think, about three percent of the overall data mix, um, and they weren't getting the levels of performance that they saw when they tried it with the Python code base, and it was obviously, like, not great for them to have a bad experience, but it was nice to be able to correlate it with the, the actual, like, Objective data mix that we saw. So we did, um, what we've been doing is like little top up fine tunes where we take like the General Genie model and do an incremental fine tune on top with just a bit more data for a given, you know, vertical language. Um, and we've been seeing improvements coming from that. So again, this is one of the great things, um, about sort of baptism by fire and letting …

AI assessment note: “we had some guys using it, um, with a C-sharp code base”

Q Yeah. I'll recommend the DynamoDB book for people who need to use DynamoDB. But we're not here to talk about AWS. We're here to talk about OpenAI. You joined OpenAI pre-ChatGPT. I also had the opportunity to join and I didn't. What was your insight?

A Yeah. I think a lot of people who joined OpenAI joined because of a product that really gets them excited. And for most people it's ChatGPT. But for me, I was a daily user of Copilot, GitHub Copilot, and I was, like, so blown away at the quality of this thing. I actually remember the first time seeing it on Hacker News and being like, wow, this is absolutely crazy. Like, this is going to change everything. And I started using it every day. It just really, I, even now when, like, I'm, I don't have service and I'm coding without Copilot, it's just, like, 10 X difference. So I was really excited about that product. I thought now is maybe the time for AI. And I had done some AI in college and thought some of those skills would transfer. Um, and I got introduced to the team. I liked everyone I talked to. So I thought, thought it'd be cool. Why didn't you join?

AI assessment note: “for me, I was a daily user of Copilot, GitHub Copilot”

Q What do you say it is you do here?

A Yeah, so yeah, I'm, I'm the tech lead for the API, but also I work on the API models team. And this team is really working on making the best models for the API. And a lot of common deployment patterns are research makes a model, and then you kind of ship it in the API. But, you know, I think there's a lot you miss when you do that. You miss a lot of developer feedback and things that are not kind of immediately obvious. What we do is we get a lot of feedback from developers, and we go and make the models better in certain ways. So our team does model training as well. We work very closely with our post training team. And so for structured outputs, it was a collab between a bunch of teams, including safety systems, to make, you know, a really great model that does, uh, structured outputs.

AI assessment note: “I'm the tech lead for the API, but also I work on the API models team.”

Q All right, let's go outside of OpenAI. What about just you personally? So you mentioned Waterloo. Maybe let's just do why is everybody at Waterloo cracked, and why are people so good, and like, why have people not replicated it, or any other commentary on your experience?

A The first is the co-op program. It's obviously really good. You know, I did six internships, learned so much in those. I think another reason is that Waterloo is like, you know, it's very cold in the winter. It's pretty miserable. There's like not that much to do apart from study and like hack on projects. And there's this big like hacker mentality, you know, there's a Hack the North is a very popular hackathon. And there's a lot of like startup in computers. It's kind of just has this like startup and hacker ethos. Then that combined with the six internships means that you get people who like graduate with two years of experience and they're very entrepreneurial and, you know, they're down to grind.

AI assessment note: “The first is the co-op program. It's obviously really good. You know, I did six internships”

Q First request. Is that an issue? Is that going to go down over time? Is there just an overhead to parsing JSON that is just insurmountable?

A It's definitely not insurmountable, and I think it will definitely go down over time. We just kind of take the approach of, of ship early and often. Um, and you know, if you, if there's nothing in there you, you don't want to fix, then you probably ship too late. Um, so I think we will get that latency down over time. But yeah, I think for most developers, it's not a big concern because you're testing out your integration, you're, you're sending some requests while you're developing it, and then it's fast and prod. So it kind of works for most people. The alternative design space that we explored It's like pre-registering your schema, so like a totally different endpoint, and then passing in like a schema ID. But we thought, you know, that was a lot of overhead, and like another endpoint to maintain, and just kind of more complexity for the developer. And we think this latency is going to come down over time, so it made sense to keep it kind of in chat completions.

AI assessment note: “It's definitely not insurmountable, and I think it will definitely go down over time.”

Q Yeah. I thought maybe we could just kind of dive into each of those actions. Code retrieval, one of the core problems you had at Indexer that You've worked on, uh, even as, as built. What makes it hard? What approach you thought would work? Didn't work? Anything like that?

A It's funny. I had a similar conversation to this when I was chatting to the guys from OpenAI yesterday. The thing is that searching for code specifically semantically, at least to start with, I mean, like keyword search and stuff like that is a, is a sole problem. It's been around for ages, but at least being able to, the phrase we always used back in the day was searching for what code does rather than what code is. Like searching for functionality is really hard, really hard. The way that we approached that problem was that obviously like a very basic and easy approach is right. Let's just embed the code base. We'll chunk it up in some arbitrary way, maybe using an AST, maybe using number of lines, maybe using whatever, like some overlapping, just chunk it up and embed it. And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work well at all because fundamentally, if you think about like semantically how code looks is very different to how English looks. And there's like not a huge amount of, of signal that's carried between the two. So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take …

AI assessment note: “Let's just embed the code base... That doesn't work.”

Q Yeah. I thought maybe we could just kind of dive into each of those actions. Code retrieval, one of the core problems you had at Indexer that You've worked on, uh, even as, as built. What makes it hard? What approach you thought would work? Didn't work? Anything like that?

A It's funny. I had a similar conversation to this when I was chatting to the guys from OpenAI yesterday. The thing is that searching for code specifically semantically, at least to start with, I mean, like keyword search and stuff like that is a, is a sole problem. It's been around for ages, but at least being able to, the phrase we always used back in the day was searching for what code does rather than what code is. Like searching for functionality is really hard, really hard. The way that we approached that problem was that obviously like a very basic and easy approach is right. Let's just embed the code base. We'll chunk it up in some arbitrary way, maybe using an AST, maybe using number of lines, maybe using whatever, like some overlapping, just chunk it up and embed it. And once you've done that, I will write a query saying like, find me some authentication code or something, embed it, and then do the cosine similarity and get the top of K right. That doesn't work. And I wish it did work. Don't get me wrong. It doesn't work well at all because fundamentally, if you think about like semantically how code looks is very different to how English looks. And there's like not a huge amount of, of signal that's carried between the two. So what we ended up, the first approach we took and that kind of did well enough for a long time was, okay, let's train a model to be able to take …

AI assessment note: “searching for functionality is really hard... Let's just embed the code base... That doesn't work”

Q Yeah. I want to give Joseph a chance to comment. Does that also mirror your, we know your story about going into, into vision, but like in the past year, since we did our podcast on Sam, um, what's been the impact that you've seen?

A Segment anything set a new standard in computer vision, you know, recapping from, from the first release to, to present. Sam introduces the ability for models to near zero shot, meaning without any training, identify kind of perfect polygons And outlines of items and objects inside images. And that capability previously required lots of manual labeling, lots of manual preparation, clicking very meticulously to create outlines of individuals and people. And there were some models that attempted to do zero shot segmentation of items inside images, though none were as High quality as Segment Anything. And with the introduction of Segment Anything, you can pass an image with SAM one, SAM two videos as well, and get perfect, pixel perfect outlines of most everything inside the images. Now, there are some edge cases across domains, and similar to the human eye, sometimes you need to say, like, which item maybe you most care about for the downstream task and problem you're working on. Though, SAM has accelerated the rate at which developers are able to use computer vision in production applications. So at RoboFlow, we were very quick to enable the community of computer vision developers and engineers to use SAM and apply it to their problems. The principle ways is using SAM. You could kind of use SAM as is to like pass an image and receive back masks. Another use case for SAM Is in pr…

AI assessment note: “SAM has accelerated the rate at which developers are able to use computer vision”

Q I cannot end this podcast without talking about the architecture because this is your effectively the, the sort of research level, architecture level innovation that enabled what I've been calling object permanence for Sam and it's memory attention. What was the inspiration going into it? And, you know, what did you find?

A Yeah, so at a high level, the way we think about extending SAM to video is that an image is just a special case of a video that just has one frame. With that idea in mind, we can extend the SAM architecture to be able to support segmentation across videos. So this is a quick video that shows how this works. So SAM architecture, we have the image encoder, we have a prompt encoder, we have a mask decoder. You can click. On an image, and that basically is a prompt. We use that prompt along with the image embedding to make a mask prediction for that image. Going to SAM-II, we can also apply SAM-II to images because we can, you know, as I said, treat an image as a video with a single frame. And so when we are in the SAM-II architecture, we introduce this new memory mechanism that consists of Three main components. There's memory attention, there's a memory encoder, and then there's a memory bank. And when we apply SAM to images, these are effectively not used, and the architecture just collapses down to the original SAM architecture. But when we do apply this to video, the memory components become really useful because they provide the context of the target object from other frames. And so this could be from the past frames, it can be from There's two types of memory. So there's like the condition, conditional frames or the prompted frames, which are basically the frames at which a …

AI assessment note: “an image is just a special case of a video that just has one frame.”

Q So as an audience surrogate who knows less than the two of you, I'm going to ask a really dumb question maybe, but is everyone using stock, a segment, anything? Are they fine tuning for the medical domain? Like how on earth could it work for the medical field without fine tuning, right? Like, is that a thing?

A So I mean, I can give a quick perspective from the research side. So one of the things, design decisions we made in SAM was to not have class labels. And so all the data is annotated in a class agnostic way. So anything that has a boundary, we consider to be an object. So for example, in any image, there's lots of small objects. We might not know what the name of them are, but you can draw a boundary around it. So you can imagine that we have eleven million images in the SA-I-B dataset. We annotated all the objects. There's many, many small objects. And so if you think about cells, there are also kind of small objects. There's probably things in the training data that looked like it, but we didn't have to label it. And so that means that even when you use SAM for applications that it wasn't really trained for, because we didn't Restricted to a certain set of categories, you can actually use it out of the box without custom adaptation. But having said that, there's probably certain domains where you need some expertise in order to be able to segment something properly. And for those use cases, having some extra fine tuning data would probably help. And we've sort of seen that there's some papers that have come out that do this. And, you know, we'd love to hear Joseph, how people are collecting data with SAM and fine tuning for their use cases.

AI assessment note: “because we didn't Restricted to a certain set of categories, you can actually use it out of the box”

Q to be used by billions of people. So what's the updated state? Of, of scaling laws. I think there was obviously the Kepler, and then there was Chinchilla, and then people kind of called the LAMA scaling law, like the 100 to 200 X kind of parameter to token ratio. What's your updated thinking on how to think about scaling laws when you pick model size and, and training data?

A Right. So, you know, as you said, this Kaplan paper with scaling laws, but they figured out, basically they tried two dimensions. The model weights, and the number of training time, like number of steps, training tokens, epochs. And for that, they figured that model size is what matters, so GPT-free was way too big compared to the actual number of training tokens, because they did a mistake not adapting the scheduler. That's what Chinchilla emphasized and discovered. To be fair, I think OpenAI knew that at the time of Chinchilla paper. But yeah, basically, Chinchilla said, We have to revisit the scanning laws originally published by Kaplan and emphasize much more the importance of training tokens. And they did like some really good scanning laws showing that there's an optimal, basically you need to double the number of training tokens every time you double the training weights to get an optimal ratio so that for a finite number of compute, you will end with the best results in your paper. And what I call the Chinchilla trap is that that's good if you want the best flagship model that obtains the highest performance on your paper. But if you want to use your model at inference time, inference, the two dimensions, one remains the model weights, but one drops the number of tokens you train it, number of steps. And so, to be compute efficient at inference time, it's much better to…

AI assessment note: “if you consider inference time, you need to go beyond Chinchilla.”

Q in the vocab, you have this kind of like concepts representatives token. So usually the larger the vocab size, the more nuanced the model can be about thinking about different things. What are the scaling laws of those organizers? You know, is a 120 a K kind of like very large and it doesn't really matter. Like, do you want to double it? Like, any thoughts there would be great.

A There's a lot of dimensions to take into account here, I think. The first thing obvious to say is Lama III compared to Lama II is multilingual, has multilingual capabilities. We worked on that. And so, because you have languages that are not just Latin languages like English, there's a lot of different characters, uh, you want to include them, uh, to represent, like, special words there, and so you need to have a bigger, uh, vocabulary size. That's the obvious thing. Which is also probably why GPT-Force Zero Uh, has a much bigger vocabulary, as it's like naturally multilingual, multimodal, uh, and speech. So, that's why we went to from 30 to 128 vocabulary size. The interesting thing, I think, to discuss about tokenizer is about scaling laws related to that. If you increase your, uh, vocab size, well, you have a bigger matrix, Which takes longer to compute. It depends on the model size. But for a small model, it has a much bigger impact than a bigger model. So, increasing that, basically saying otherwise, the number of vocabulary size, 428, is the same than the eight, 70, or four zero five b. But so, relatively in percentage of the total number of weights, for the seven b, it's much more than the four zero five b. Well, it's small compared to total number of weights. So, that has more impact in terms of training speed. There. But what is interesting is, with a bigger vocabulary…

AI assessment note: “The interesting thing, I think, to discuss about tokenizer is about scaling laws related to that.”

Q so many different usage of this thing, but specifically synthetic data for pre-training, it feels almost like you're running multiple epochs On the raw data when it's, while it's rephrased or reformatted by a language model, right? And in my mind, it's very similar to computer vision, where you do data augmentation on an item, right? Like we're, we're doing data augmentation. That's the less cool name for synthetic data.

A That's very interesting. I totally agree with you related to pre-training, totally a standpoint you said. I think it's very different though for post-training in the future direction on synthetic data that I'm personally excited. Like, for instance, what I'm excited about is we had this survey on augmented LLM a year ago, and all the idea is, like, if you augment your LLM with something else, it can be a retriever, it can be search, it can be a tool, it can be a calculator, it can be a code execution. Then you are not just distillating, like, doing some data augmentation with your model, But you're actually adding some expert skills that possibly goes beyond the model weights. For instance, if your model, like, can calculate something it was wrong before, and now it has access to a calculator, and you can retrain your model on that, then you're learning something new. If your model didn't know something about LAMA two probably doesn't know a lot about LAMA three, but now if it can search online about it, and then you train the model on that, Then you have a positive feedback loop, like what we call expert direction, targeting directly the weakness of the model. It's like continual augmentation of the language model, much beyond just that augmentation.

AI assessment note: “I totally agree with you related to pre-training, totally a standpoint you said.”

Q and like looking at all these new, uh, Hit new headlines is, is really helpful. But then, um, it only gets you, like, a very surface level understanding. Then you still need a process to decide which one to invest in. Um, so I'm, I'm trying to dig for, like, what is your formula for, like, deciding, you know, what to go deep on and what to kind of skip?

A From a practical standpoint, as a company, like, I already know the, there are, like, three to five Things that will be valuable and useful to us, and then there's other stuff that's like out of scope from, from, for different reasons. Some stuff is like out of scope from, um, hey, this is not going to impact or help us, and then other things are out of scope because we can't do it. You know, like the, the stuff, like different tech, so a really good instance for that is, um, specific algorithms for, um, you know, Improving extremely large-scale distributed training. Like, that's that, we're not gonna have the opportunity to get 2000 each 100. If we do, it'd be really cool. But, like, I'm just saying, like, as for now, like, you gotta, you gotta reach for the things that would be useful. Things that would be useful for us, for instance, um, are, for everybody, actually, to be honest, is like, um, evaluations, uh, different post-training techniques. And then synthetic data, uh, uh, construction. Like, we're always on the, I'm always on the look for that, and then how do I figure out whether these things, um, you know, which new piece of news is actually novel? Um, well, that's sort of my, like, mental cache to a certain extent. Like, I've built up, like, this state of, like, I already know, like, all the things that have already been written, uh, for the state of the art, uh, fo…

AI assessment note: “I already know the, there are, like, three to five Things that will be valuable”

Q can set your own lyrics, you can set your own rhythm, you can set the title of the song and whatnot. What are, how do you see users distribute themselves? You know, I'm guessing a lot of people use the easy mode. Like, are you seeing a lot of power users using the custom mode and maybe some of the favorite use cases that you've seen so far on Suno?

A Yeah, actually, um, more than half of the usage is, uh, that expert mode. And people really like to get into it and start tweaking things and adding things and playing with words or line breaks or different ad lib and people really love it. It's, it's, it's really fun. So I think, you know, there's kind of two modes that you can access now. One is that single box where you kind of just describe something, and then the other is the expert mode, and, um, those kind of fit nicely into two use cases. The first use case is what we call nice shit posting, and it's basically like something funny happened, and I'm just going to very quickly make a song about it, and the, the example I I'll usually give is like, I walk into Starbucks with one of my co-founders, he gives his name Martin, his coffee comes out, Um, with the name Margu and I can in five seconds make a song about this and it has immortalized it and that Margu song is stuck in all of our heads now and it's like funny and light and there's levity that you've brought to that moment and the other is that you got just sucked into I need there's this song that's in my head and I need to get it out and I'm going to keep tweaking it and listening and having ideas and tweaking it until I get the song that I want and um Those are very different use cases, but I think ultimately there's so much in between these two things that it's jus…

AI assessment note: “more than half of the usage is, uh, that expert mode.”

Q is that, uh, like, what did, what did Bark lean off of? Like, uh, because obviously I think there was a lot of preceding TTS work that was in open source. Um, how much of that did you use? How much of that was, like, sort of brand new from, from your research? Um, what's the intellectual lineage, uh, there, just, just to cover out the, the speech recognition style?

A So it's not speech recognition. It's, it's text to speech. But, um, as far as I know, um, there was no other, uh, certainly not in the open source, uh, text to speech that was kind of transformer based. Everything else was what I would call the old style of doing things where you build these kind of single purpose models that are really good at this one narrow task. And you're kind of always data limited and the availability of high quality training data for text to speech, um, is limited. Um, I don't think we're necessarily all that inventive to say we're going to try to train in a self supervised way, a transformer based model that on kind of lots of audio, uh, and then kind of tweak it so that we can do text to speech based on that. That would be kind of the new way of doing things in a foundation model is the, is the buzzword, if you will. And so, you know, we built that up, I think from scratch, a lot of shout outs have to go to lots of different things, whether it's Uh, papers, but also, uh, it's very obvious. Uh, there's a big shout out to, um, Andre Karpathy's nano GPT. Um, you know, there's a lot of code borrowed from there. Um, I, I think I, we are huge fans of that project. It's just to show people how you don't have to be afraid of GPT type things. And it's like, um, yeah, it's actually not all that much code to make performant transformer based models. And, you kno…

AI assessment note: “a big shout out to, um, Andre Karpathy's nano GPT”

Q Yeah. Yeah. Um, and, and then, um, so that was the start of your data career. You also wrote a couple of popular sort of open source, uh, tooling, uh, while, while you were there. Um, and then, and then you, is that, is that correct or?

A No, that's right. I mean, I was at Spotify for seven years. This is a long stint. Uh, and, and Spotify was a wild place early on. And I mean, the data space is also a wild place. I mean, it was like Hadoop cluster in the like foosball room on the floor. Um, and you know, so, so like it was, It was a lot of crude, like, very basic infrastructure, and I didn't know anything about it, and, and, like, I was hired to kind of figure out data stuff, and I started hacking on a recommendation system, and then, you know, got sidetracked into a bunch of other stuff. I fixed a bunch of reporting things, and, and set up A-B testing, and started doing, like, business analytics, and later got back to music recommendation system, and a lot of the infrastructure didn't really exist. Like, there was, like, Hadoop back then, which is kind of bad, and I, I don't miss it, but spent a lot of time with that. Uh, as a part of that, I ended up building, uh, Workflow engine called Luigi, which is like briefly like somewhat like widely ended up being used by a bunch of companies. Sort of like, you know, kind of like Airflow, but like before Airflow, um, I think it did some things better, some things worse. Uh, I also built a vector database called Annoy, which is like for a while it was actually quite widely used, uh, in 2012. So it was like way before like all this like vector database stuff ended up ha…

AI assessment note: “I ended up building, uh, Workflow engine called Luigi”

Q Is there something you're focused on this year? Hiring, building, engineering team. What should people that want to work together know?

A You know, all those things. I, I, I think, um, hiring is, uh, A pretty big topic. We are, um, 38 people on the team, um, and we are hiding across all the areas, um, you know. Uh, CUDA and Kernel Hacker, we have lots of exciting projects. Um, if you're a researcher, you like to build models, we have exciting projects. If you work on systems and infrastructure and the cloud layer, um, you know, we, we do a lot of work there. And, uh, as, as well as sort of front end and developer experience and applications. Uh, so really kind of across the board, we have, I think, 20 plus postings on our, Uh, job openings on our site, uh, and, and folks who are passionate about open and, you know, AI, um, I also say if you, if you, you know, people looking at together, they don't necessarily, for all the postings, have to have experience, um, you know, professional experience working in machine learning or AI, um, Uh, many of the systems people are sort of doing this for the first time, and they can apply their, uh, you know, systems expertise to, um, to the kind of things that we are doing, and we can, we can teach people AI as long as they have expertise in other areas.

AI assessment note: “hiring is, uh, A pretty big topic. We are, um, 38 people on the team”

Q was, like, quite a bit higher. So just, I think that kind of progress over time is what we're most interested in seeing is, you know, are models getting worse? Model's getting better? Are people still loving PG vector? Do people still love Mongo? You know, stuff like that. That I think is the most interesting thing, so. Do you two have any questions that you think we should ask?

A Um, off the bat, like, it's, it seems like you're very, uh, language model focused. Um, um, you know, I think that there's an increasing, um, interest in multi-modality, um, in, in AI. Um, and I don't really know how that is going to manifest. Um, obviously, GPT, Four Vision, as well as, um, Gemini both have multi-world capabilities. Uh, there's a smaller subset of open source models that have, Um, multimodal features as well. Like I, we just released an episode today, uh, talking about IdaFix from HuggingFace. And, uh, yeah, so I think I, I think I w I would like to understand how people are adopting or adapting to the different modalities that are now coming online for them. Um, what their demand is relative to, uh, for like, let's say generative images versus, um, you know, like just visual comprehension versus, um, audio versus, uh, text to speech. Like, uh, what do they want? What do they need? And what's the sort of, um, yeah. Forced, like, stack ranked, um, preference order. Yeah. Yeah.

AI assessment note: “I would like to understand how people are adopting or adapting to the different modalities”

Q How did that, like, can we tell that story, like, so how did this happen, right? Like, so, Uh, you, you were inspired by, by Grok.

A Yeah, so, I guess, like, you know, the backstory from my point of view is I had used CodeSearch and Grok while at Google, um, but I, I didn't actually know that it was connected to you, Steve. Like, I knew, I knew you from your blog posts, which were always, like, excellent, kind of, like, inside, very thoughtful takes on, uh, from an engineer's perspective on, on some of the challenges facing, like, tech companies and, you know, tech culture and that sort of thing. Um, But my first introduction to you within the context of, like, code intelligence and code understanding was I watched a talk that you gave, I think, at Stanford about Grok when you were first building it, and that was very eye-opening. And I was like, oh, like, that guy, like, the guy who, you know, writes the extremely thoughtful, ranty, like, blog posts also built that system. Um, and so that's, that's how, that's how I knew, you know, you were kind of involved in that. And then it was kind of like, uh, you know, we always, Kind of like wanted to hire you, uh, but never knew quite how to, uh, approach you or, you know, get that, get that conversation started.

AI assessment note: “I had used CodeSearch and Grok while at Google... watched a talk that you gave”

Q Um, so would you basically say that they're tackling the agent's problem that you don't want to tackle?

A No, I would say at a high level, we are after, uh, maybe like the same high level problem, which is like, hey, I want some code written, I want to develop some software, uh, and can, can an automated system go build that software, uh, for me? Um, I think the, the approaches might be different. Um, so, I think the analogy in my mind is, I think about like the AI chess players, right? Like, is, like, Coding in some sense is, I mean, it's similar and dissimilar to chess. Uh, I think one question I ask is like, do you think producing code is, is more difficult than playing chess or less difficult than playing chess? More? I think more, right? And, and if you look at, like, the, the best AHS players, like, yes, you can use an LLM to play chess. Like, people have showed demos where it's like, oh, like, yeah, uh, GPT-IV is actually a pretty decent, like, chess move suggester, right? Um, but you would never build, like, a best-in-class, uh, chess player off of GPT-IV, uh, alone, right? Like, the way that people design, uh, chess players is that you have, kind of, like, a search space, uh, and then you have, Uh, a way to explore that search space efficiently. There's a bunch of search algorithms, essentially, where you were doing tree search in various ways, and, uh, you can have heuristic functions which might be powered by an LLM, right? Like, you might use an LLM to generate proposal…

AI assessment note: “No, I would say at a high level, we are after, uh, maybe like the same”

Q that software developer that is not no longer doing these routine boilerplate type tasks because they're just enhanced by tools like yours. And so you, you mentioned, uh, you open code graph. I mean, that, that is a kind of, Uh, DSL maybe. And, um, because we're releasing this, uh, as you, as you go GA, um, you hope to, uh, for other people to, to take advantage of that?

A Oh yeah, I would say, so OpenCodeGraph is not a DSL, it's more of a protocol. It's basically like, hey, if you want to make, uh, your system, whether it's, you know, chat or logging or whatever, accessible to, um, an AI developer tool like Kodi, um, here's kind of like the, the, the schema. Uh, by which you can provide that context and offer hints. Um, so I would, you know, comparisons like LSP obviously did this for, uh, kind of like standard code intelligence. It's kind of like a lingua franco for providing find references and code definition. There's kind of like analogs to that. There might be also analogs to, uh, kind of the original OpenAI, kind of like plugins, uh, API, where it's like, hey, you know, there's, there's all this like context out there that might be useful for, uh, an LM based system to consume. Uh, and so, at a high level, what we're trying to do is, uh, define a, a common language, uh, for context providers to provide context to other tools in the software development lifecycle.

AI assessment note: “OpenCodeGraph is not a DSL, it's more of a protocol.”

Q your, your AI stack? Um, what do you, like, obviously you build a lot proprietary in-house, um, but, like, what approaches, uh, you know, like, so prompt, prompt engineering, do you have a prompt engineering management tool? You know, what, what approach is there? Do you, do you do, um, pre-processing, pre-processing orchestration, like, do you use Airflow? Do you use something else? Like, you know, that kind of stuff.

A Yeah, ours is very, like, duct taped together at the moment. Um, so in, in terms of stack, uh, I mean, it's essentially, uh, Go and TypeScript, uh, and now Rust. Um, there's the, the knowledge graph, the code knowledge graph that we built, which is using indexers, uh, many of which are open source, um, that speak the skip protocol. Uh, and, uh, we have the code search backend, um, You know, traditionally we've supported regular expression search and, uh, uh, string literal search with like a trigram index, and we're also building more like fuzzy search on top of that now, uh, kind of like natural language or keyword based search on top of that. Um, and we use a variety of open source and proprietary models. We try to be like pluggable with respect to different models so we can easily kind of like swap, swap the latest model in and out, uh, as they come online.

AI assessment note: “in terms of stack, uh, I mean, it's essentially, uh, Go and TypeScript”

Q Fantastic. Um, I, I, I think we might want to give people, uh, some impression about, like, the type of traffic that you have, um, because when you present it with a text box, you could type in anything, and I don't know if you have some mental categorization of, like, what are, like, the top three use cases that people tend to coalesce?

A Yeah, that's a great question. Um, so The two main types of searches that we see are how-to questions, like how to do x using y tool. Um, and this historically has been our bread and butter because, uh, with our embeddings, like, we're really, really good at just going over a bunch of developer documentation and figuring out exactly the part that's relevant and just telling you, okay, like, you can use this method. But as LLMs have gotten better, And as we've really transitioned to, um, using GPT-IV a lot in our product, um, people organically just started pasting in code that's not working and just said, fix this. Yeah. And what really shocks us is that, um, a lot of the people who do that, um, they're coming from ChatGPT. So they tried it in ChatGPT with ChatGPT-IV. It didn't work. Uh, maybe it required like some multi-step reasoning. Maybe it required, um, to like, Some internet context or something found in either a Stack Overflow post or some documentation to solve it. Um, and so then they paste it into find and then find works. Um, so those are really those two different cases. Like, how can I build this conceptually or like remind me of this one detail that I need to, to build this thing or just like, here's this code, fix it. Um, and so that's what a big part of our VS Code extension is, is like enabling a much smoother here, just like fix it for me. Type of workflow. T…

AI assessment note: “The two main types of searches that we see are how-to questions”

Q I'm on the roster. Thank you so much. Um, what, why, so why spend time being a VC? Uh, and, and organizing all these events. You're also a very busy CEO and, you know, why, why spend time with that? Why is that an important part of your life?

A Yeah, for me personally, I really like helping founders. So, um, Ali, my, uh, investing partner, is fortunately amazing, and she does everything for the fund. Um, so she, like, hosts the Thursday night events, and she finds, uh, folks who we could invest in, and she does basically everything. Josh and I, um, are her co-partners. So Ali was our former chief of staff at Sorceress, and we just thought she was amazing. Um, and she wanted to be an investor, and Josh and I also Like care about helping founders and kind of like giving back to the community. What we didn't realize at the time when we started the fund is that it would actually be incredibly helpful for imbue. So, uh, talking to AI founders who are building agents and working on, you know, similar things is really helpful. They could potentially be our customers and they're trying out all sorts of interesting things. And I think being an investor, looking at the space from the other side of the table, it's just a different hat That I routinely put on and it's helpful to see the space from the investor lens as opposed to from the founder lens. Um, so I find that kind of like hat switching valuable. It maybe would lead us to do slightly different things.

AI assessment note: “for me personally, I really like helping founders... helpful to see the space from the investor lens”

Q of your first goals is like build the framework grind time and driver for, for AMD. Uh, and then on, On June third on Twitch, uh, you weren't as excited about AMD anymore. Maybe let's talk a bit about that. And like, uh, you compared the quality of like commit messages from like the AMD kernel to like the Intel work that people are doing there. What's important to know?

A So when I said I wanted to, I want to write a framework, I didn't never intended on writing a kernel driver. I mean, like I flirted with that idea briefly, but like realistically, I like there's three parts to it, right? There's like the ML framework, there's the driver, and then there's the user space runtime. I was even down to rewrite the user space runtime. I have, I have a GitHub repo called CUDA IO control sniffer. It's terribly called, but you can actually launch a CUDA kernel without CUDA. So you don't need CUDA installed, just the Nvidia open source driver. And this open source repo can launch a CUDA kernel. So rewriting the user space runtime is doable. Rewriting the kernel driver? I don't even have docs. I don't have any docs for the GPU. Like, it would just be a massive reverse engineering project. Um, so that is, when I saw that there, like, it wasn't, like, I wasn't complaining about it being slow. I wasn't complaining about PyTorch not compiling. I was complaining about the thing crashing my entire computer. It panics my kernel, and I have to wait five minutes while it reboots because it's a server motherboard, and they take Like five minutes to reboot. Um, so I was like, look, if you guys do not care enough to get me a decent kernel driver, there's no way I'm wasting my time on this, especially when I can use Intel GPUs. Intel GPUs have a stable kernel driver an…

AI assessment note: “I was complaining about the thing crashing my entire computer. It panics my kernel”

Q What's the intuition between why you need to do it in the model?

A My intuition behind the actual, when do you train or even post train a model versus just prompt a model is if the model has to learn the underlying physics of the world that it's operating in. So it has to learn new social physics. The places where it doesn't have to train is it already has the physics. We trust the physics. It already has the base statistics, but it's just trying to react to an environment. Then I think you can just prompt your way into getting the, you know, actions out of it. I don't think the model has yet, at least the models that are out in the open, has yet learned the complete mapping of social physics of humanity. Uh, this actually is one of the core thesis of simile, right? And one of the core reason why that is the case is if you look at the data that the model was trained on, these models were trained on the web data and whatever was available on the web. And these are really interesting data sets, but they are fundamentally the self exposed attitudinal data with some behavior data that's sprinkled around here and there. And It has yet to learn really deep behavioral nature of people. Not just what people say they do online, but they, what they actually do in real life. And this is actually one of the sort of, ah, what I would consider to be the dark knowledge of humanity that we haven't quite captured. And it's these kind of data that would also ne…

AI assessment note: “if the model has to learn the underlying physics of the world that it's operating in”

Q Yeah. Are there other case studies? So you talked about CVS, talked about Gallup, Deloitte, Worldfront?

A Worldfront is an interesting one, um, because one of the things they were trying to do, they were one of the first customers that wanted to actually do product testing. That goes beyond just asking people what they think about, let's say, behavior experiments and so forth. So there, really what we Had to do was reason about multimodal input. So images, but also you can also imagine like these agents traversing through Figma mockups or websites. So some of the things that our agents can also do is you can be given a domain, like a website URL and actually go use it for a while. It's these kind of things. And Wealthfront was one of the first, uh, customers, uh, that was very excited about this possibility.

AI assessment note: “Worldfront is an interesting one, um, because one of the things they were trying”

Q I think that's, that's super clear. And then also the other thing I wanted to dive into was your, uh, the productivity team. Uh, what else is there? First of all, you know, what, what, what are the top level teams other than productivity? Isn't productivity everything?

A So, you know, we have a team focused on, uh, on ChatGPT, like the, the core chat experience, um, for consumer, which is like, you know, not, I think all productivity, like there's people are using ChatGPT every day for search to, you know, figure out how to write messages to loved ones, to think about Um, how to like learn a new topic, et cetera. And so there's so much more inside to create images. There's so much more in chat that, you know, the hundreds of millions of users are using that, um, you know, obviously that, that warrants like a very dedicated effort. Um, and there's teams focused on enterprise and infrastructure and API and stuff like that as well.

AI assessment note: “teams focused on enterprise and infrastructure and API and stuff like that as well”

Q Cool. Uh, you were gonna, you lead the productivity team. How do you define productivity?

A I think our mission is to make it possible for people to do things that they weren't able to do before. And right now we're thinking about it from the perspective of knowledge work. And so when I look at knowledge work, I think about people are no longer siloed by their roles. They're no longer siloed by maybe the, the, um, background or training that they have. Like, no matter what function you're in, you can suddenly build things. It's only get access to data that you otherwise might not be able to interpret, et cetera. Um, and then I think that extends to your personal life where we want to give you leverage at the end of the day. Like we want the models and the product to be able to give you leverage so that you can, you know, create time for yourself to do the things that you love.

AI assessment note: “make it possible for people to do things that they weren't able to do before”