The Exchanges

Q Fantastic. Um, I, I, I think we might want to give people, uh, some impression about, like, the type of traffic that you have, um, because when you present it with a text box, you could type in anything, and I don't know if you have some mental categorization of, like, what are, like, the top three use cases that people tend to coalesce?

A Yeah, that's a great question. Um, so The two main types of searches that we see are how-to questions, like how to do x using y tool. Um, and this historically has been our bread and butter because, uh, with our embeddings, like, we're really, really good at just going over a bunch of developer documentation and figuring out exactly the part that's relevant and just telling you, okay, like, you can use this method. But as LLMs have gotten better, And as we've really transitioned to, um, using GPT-IV a lot in our product, um, people organically just started pasting in code that's not working and just said, fix this. Yeah. And what really shocks us is that, um, a lot of the people who do that, um, they're coming from ChatGPT. So they tried it in ChatGPT with ChatGPT-IV. It didn't work. Uh, maybe it required like some multi-step reasoning. Maybe it required, um, to like, Some internet context or something found in either a Stack Overflow post or some documentation to solve it. Um, and so then they paste it into find and then find works. Um, so those are really those two different cases. Like, how can I build this conceptually or like remind me of this one detail that I need to, to build this thing or just like, here's this code, fix it. Um, and so that's what a big part of our VS Code extension is, is like enabling a much smoother here, just like fix it for me. Type of workflow. T…

AI assessment note: “The two main types of searches that we see are how-to questions”

Q Was this a precursor to Bloom? Because Bloom's the one that I know.

A I think Bloom ended up actually coming out in twenty-twenty-two, but Bloom had other problems where I think, for whatever reason, the Bloom models just were never really that good, which is so sad because I really wanted to use them, but I think they didn't turn on that much data. Um, I think they used, like, the original, they were trying to replicate GPT-III, so they just used those numbers, which we now know are, like, far below Chinchilla Optimal, and even Chinchilla Optimal, which we can, like, talk about later, like, what we're currently doing with the fine model goes, yeah, it goes way beyond that. Um, but they weren't sharing enough data, I'm not sure how that data was clean, but it probably wasn't super clean, and then they didn't really do any fine-tuning until much later. Um, so T-zero worked well because they took the T-five models, which were, um, closer to Chinchilla Optimal, because I think they were trained on also like 300 something billion tokens similar to GPT-three, but the models were much smaller. Um, uh, so the models, yeah, they were pre-trained better, and then they were fine-tuned on, um, This is, I think T-Zero is the first model that did large-scale instruction tuning, um, from diverse data sources in the fall of twenty-twenty-one. Um, this is before InstructGPT. Um, this is before Flan T-Five, which came out in twenty-twenty-two. This is, I think, t…

AI assessment note: “I think Bloom ended up actually coming out in twenty-twenty-two”

Q You didn't say how much? It was just, like, just give them GPUs.

A Basically, yeah. Ron is known for Writing these, like, one-liner emails that are, like, very short but very to the point, and I think that's why, like, like, everyone responds to Ron. Everyone loves Ron, um, and so Jensen responds. He responds quickly, like, tagging this VP of AI NVIDIA, um, and we start working with NVIDIA, which is great, um, and something that I love about NVIDIA, by the way, is that after that intro, um, we got matched with, like, a dedicated team, and, um, at NVIDIA, They know that they're going to win regardless. So they don't care where you get the GPUs from. They're like, they're truly neutral, unlike various sales reps that you might encounter at various like clouds and, you know, hardware companies, et cetera. Like they actually just want to help you because they know they don't care. Like regardless, they know that if you're getting NVIDIA GPUs, they're still winning. So, um, so I guess that's, that's a tip, um, is that like, if you're looking for GPUs, like NVIDIA, yeah, they're, they'll, they'll help you do it.

AI assessment note: “Basically, yeah. Ron is known for Writing these, like, one-liner emails”

Q So Bing web search, uh, and then, and then in August you launched V-II. Could you, um, is, is, is V-II the system, the find as a system pitch? Or have you moved, evolved since then?

A Yeah. So I don't, I, I like the V-II moniker, like, I don't really think of it that way in my mind. There's like, there's the version we launched during last summer during YC, which was, um, the Bing version directed towards programmers. Um, and that's kind of like, that's why I call it like the first incarnation of what we currently are, because it was already directed towards programmers. We had like a code snippet search built in as well, because at the time, you know, the models we were using weren't good enough to generate code snippets. Even GPT, like the text DaVinci too, which was available at the time. Wasn't that good at generating code and it would generate like very, very short, very incomplete, um, code snippets. And so, um, we launched that last summer. Got some traction, but really, like, we were only doing, like, I don't know, maybe, like, 10,000 searches a day. Like, some people knew about it. Some people used it, which is impressive, because looking back, the product, like, was not that good. Um, and yeah, every time we've, like, made an improvement to, um, the way that we retrieve context, uh, through better embeddings, more intelligent, like, HTML parsers, um, and importantly, like, better underlying models. Um, yeah, I would really consider every kind of iteration after that when we, every major version after that was when we introduced a better underlying …

AI assessment note: “I don't really think of it that way in my mind... every major version after”

Q Give people tips, right? Like, where do you, what fire hose do you drink from?

A Yeah, exactly. So like, whenever I see something that blows my mind, the way that that initial hugging face demo did, That was like the start of everything. I'll just, yeah, I'll just like, I'll start from the beginning. I'll like, if I don't know anything, then like, I'll just, I'll start by Just trying to get a mental model of what is happening. Like, first, I need to understand what, so I can understand, like, the why, the how and the why, and once I can understand that, then I can make my own hypotheses, um, about, like, okay, here are the assumptions that the authors of this made, um, and here's why maybe they're correct, maybe they're wrong, and here's how, like, I can improve on it, um, and iterate on it, and I guess that's the, the mindset that I approach it from, is, like, once I understand something, like, how can it be better? How can it be faster? How can it be, like, more accurate? Um, and so I guess for anyone starting now, like, I would have, I would have used Find if I was starting now, because, like, I would have loved to just have been able to say, like, hey, like, I have no idea what I'm doing. Can you just, like, be this, like, technical research assistant and kind of hold my hand and, like, ask me clarifying questions and, like, help me, like, formalize my assumptions, like, along the way. I would have loved that. But yeah, I just kind of did that myself.

AI assessment note: “for anyone starting now, like, I would have, I would have used Find”

Q People. Um, it's a pretty simple concept. Like what's the source of error?

A Exactly. And I was, I've been talking to Harrison actually about like a more like structured way, perhaps within Langchain to like two evals. Cause I think that's a massive problem. Like every single eval is different, uh, for these big large language models. Um, and doing them in a quantitative way is really hard. Um, but it's possible with, with like a platform that I think harnesses GPT-IV in the right way. Um, that, and also, um, perhaps a, Stricter prompting language, like a prompting markup language for prompting models is something I'm also very interested in, um, because we've written some very, very complex prompts, particularly for a VS code extension, um, to like, to like very fancy things with people's code, and like, I wish there was a way that you could have like a more formal way, like a Python for LLM prompting, that you could Activate desired things within, like, the model's execution flow through some other abstraction above language that has been, like, tested to do that some of the time. Perhaps, like, combined with, like, formal grammar limitations and stuff like that.

AI assessment note: “I've been talking to Harrison actually about like a more like structured way”