The Exchanges

Q Um, I think, uh, you know, Sophia has an opportunity to be really, um, really impactful. Uh, you started a company last year taking leave from Stanford. Um, given your work has been like theoretical, but with practical applications, like what drove you to do that?

A I think I came to Stanford partly because there's a, um, very strong industry connection here at Stanford. Compared to some of the other universities. Um, and, and also probably entrepreneurship is just part of my, um, uh, my career plan, uh, anyways. And, uh, in terms of the timing, I felt that this is the right timing in the sense that, um, the, the technologies are more and more mature so that it seems that the commercialization is the, is the right timing right now. So for example, I think, um, uh, one, one story I have is that, you know, I, I look up some of my, Um, slide stack, uh, for my, uh, lectures at Stanford CS two and nine, seven years ago, uh, when I started to teach at Stanford. Um, at that point, machine learning, uh, we have a lecture with Chris Ray and the machine learning, uh, on applied machine learning. So how do you apply machine learning industry? And there are seven steps there. So, um, the first step is you define your problem. The second step is you collect your data, um, and you choose the loss function, you train it and you iterate so and so forth. So it's pretty complicated at that point. Um, and now the foundation model, um, uh, arrives to power and, uh, and in a new foundation model era, the only thing you have to do is that you have to, um, you know, someone will tune a foundation model for you, and then, uh, you tune a prompt and you add, uh, uh…

AI assessment note: “that's why I felt that this is probably the right time to commercialize”

Q in a more general way. And so the application of, um, of AI in industry is just much, much cheaper, right? Cause you only do, you know, the last few steps and, or different set, but last few steps in essence. So maybe you can talk about like the, the, you know, just given wide range of research, the problem you focus on with voyage that you saw with customers.

A Yeah. So with, with Voyage, I think we are mostly building, uh, these two components, uh, Rerank and Embeddings for improving the quality of the retrieval or the search system. So the reason why we focus on this is because we talked to so many customers and we found that, uh, right now, um, uh, for implementing Rack, um, the bottleneck seems to be that, you know, it's, it's not very hard to implement it, right? You can just connect the components and have your Rack system ready very quickly. But the bottleneck seems to be the quality. Of the response and the quality of the response, uh, is heavily affected or is kind of almost, almost bottlenecked by the quality of the retrieval part. If the large language model, uh, see very relevant documents, then they can synthesize very good answers. Uh, even like a Lama can do that very well.

AI assessment note: “we talked to so many customers and we found that... the bottleneck seems to be”

Q is obviously going to be needed. And it's the questions are, are to me are really like, you know, does, does efficiency matter both from a cost perspective and a speed, like a latency perspective, right? And how much can you push the context window? And like, you know, does hallucination management matter? And so I, I think there are lots of arguments for like RAG being very persistent here.

A Yeah, yeah, exactly. And just to add a little bit on that. So, uh, one million tokens, five books, right? So, but many companies has a hundred million tokens. That's a hundred X difference, right? So a hundred X, you know, for cost is a, is a big difference. That could be just, uh, um, you know, a hundred K dollars versus like ten million dollars, right? Ten million dollars is Unacceptable, but a hundred K sounds okay. Yeah. I think that's probably what's going to happen. Like, so, so from, at least for many of the companies, right? So right now, if they have a hundred million tokens, I don't think they can use long context transformers at all because it's way too expensive.

AI assessment note: “a hundred X, you know, for cost is a, is a big difference.”

Q Um, can you talk a little bit about just, um, like the intuition for how fine tuning or domain specific embeddings improves performance?

A Yeah. Fine tuning and domain specific embedding models are what we are very good at at Voyage. So just to have some context here. So, uh, what we do is that we start with a, uh, um, general purpose based embedding model, which is also what we trained from scratch. Uh, and from there we, um, uh, first, Fine tune or continue pre-tune whatever you call it, uh, on, um, some domain specific data. So for example, we, uh, fine tune on two trillions of code snippets, tokens, and then we get a code embedding model and we do the, uh, uh, fine tuning on one trillion legal tokens. And that's, uh, how we got the legal embedding model and this domain specific embedding models. I didn't use any of the preparatory data so that everyone can use them, But they really excel in one particular domain and the performance in other domains are not, uh, changed much. And the reason why we do this is because the number of parameters in the embedding model is a limited. So, um, because, um, you only have like a, you have a latency budget, uh, something like maybe 1:02, sometimes like a 200 milliseconds, you know, some people even want 50 milliseconds. Um, and then, um, basically it's, uh, it's impossible to use more than ten billion parameters. For embedding models. And we have limit parameters. Any customization is very important because the customization means that you use the limit number of parameter…

AI assessment note: “customization means that you use the limit number of parameters on the right tasks”

Q just, um, zooming out, like, you know, you started by saying in order to have the debate about RAG versus alternative architectures for working on proprietary data, you need to predict forward, right? Um, any predictions for how these systems change as LLMs improve dramatically, right? If we look at the next generations of open AI and, um, or GPT and Claude and the Mistral models and Lama and such.

A Yeah. So my prediction is that the, the, the system will be simpler and simpler. Maybe this is, uh, my biased view. Um, so, or at least this is something that we are working towards. Um, so the idea of what would be that, um, it's a very, very simple system. So you just do, you just have three components like large English model, um, vector database and embedding models, and maybe four components, another ranker, um, uh, which refine the retrieved results. Um, and you connect all of this and each of the new artworks does everything else. Uh, you don't have to worry anything about trunking, multi-modality, changing the data format, um, because new artworks can do most of them, right? So seven years ago, if you talk to any of the so-called language models, seven years ago, you have to turn the format into a very, very clean format. Um, and now you talk to GPT-IV, you can have typos, you can have all kind of like a Weird formats. You can even dump JSON files to it. Um, right. So the same thing would happen for embedding models as well. So my vision is that in the future, AI will just be that, uh, a very simple software engineering layer on top of, of, of a few, um, uh, very strong neural network components.

AI assessment note: “So my prediction is that the, the, the system will be simpler and simpler.”

Q Where are you seeing, ah, RAG applications today? Like, what are, what are customers building, or, you know, what are the most common systems?

A Yeah. So we have a lot of users and, uh, and they are all over the, uh, places. I think, uh, you know, uh, we have even a customer who is a chemistry company who is building this react system to understand their, uh, chemistry, uh, documents or products in descriptions. Um, and, um, I think just, uh, it's almost everywhere, like finance, legal, code retrieval, code generation, so on and so forth. I think it can be applied to almost any cases and also for, Even for individual users, uh, where you have, you know, a lot of like individual, um, um, uh, personal information, um, and, uh, you want to have a rec system on a phone so that, uh, you can access your past information, uh, much in a much more easy way. And, um, and you want to retrieve, for example, we all have seen that, you know, when you search your, um, um, documents on your laptop, it's actually pretty hard. You have to use the exact file name. It will be much easier if, uh, this search can be semantic based.

AI assessment note: “chemistry company who is building this react system to understand their, uh, chemistry”