The Exchanges

Q into, uh, RAG, uh, and the architecture in some detail, and maybe into, I guess, the one dot O version, and then we'll spend time talking about two dot O and Gentic RAG and what contextual, uh, does, uh, but, uh, maybe as, uh, as an introduction slash deep dive, uh, how does that work? So this, uh, retriever, there's a generator, all those good things. What's the core architecture?

A Yeah, the core architecture is very simple. You have a language model, so that's the G, And then you want to give that context, and the way you do that is by augmenting it, the A, uh, using retrieval, the R. So that's R-A-G-RAG. Um, and, and so how you do the retrieval, that has been changing constantly over time. Um, so in the initial paper, we used a vector database or a face. So the, the, the words, uh, vector database didn't exist at the time. Um, but so face was the first vector database. Um, and Uh, I think people over time have started figuring out that that has all kinds of limitations, right? So how a vector database works is you, you just have embeddings. So you encode pieces of information or chunks of documents, you encode them as a vector, and then you do basic dot product similarity search. Um, but, uh, that, that has issues where you're just looking for chunks that are similar to the question, but you don't necessarily want to find chunks that are similar to the question. You want to find chunks that are relevant to answering the question, right? So you, you need to do different things with the representations, with the embeddings. So a lot of modern RAG deployments are very different from the original, uh, ideas in the paper, right? So you still have a vector database, but the way you encode things is, is very different. Um, then, uh, you usually also have a spa…

AI assessment note: “Yeah, the core architecture is very simple. You have a language model, so that's the G”

Q Yeah, and it's, it's funny how the things happen, right? This, uh, tends to be, uh, something in the water at any point in time when, like, different groups of very smart people in different organizations do think similarly. I think I heard you somewhere talk about, um, how when Transformers came out from, from your perspective, that was actually sort of underwhelming. Can you go into that quickly?

A Yeah, I mean, so, uh, it's just fun how, like, how, like, history is sort of Rewritten over time, I guess, by, like, PR departments of large enterprises. Uh, but, but, so when the, when the Transformers paper came out, um, so at FAIR we were working on very similar ideas, obviously, I guess, a bit more in the, like, convolutions space because, because of Jan LeCun's background, and I guess, like, you know, that's, that's, uh, what a lot of people were also exploring at the time, but the idea of the Transformers paper is, is really just, Can I cut the recurrence of the RNN at the time we had RNNs, right? Can we cut the recurrence because then we can do parallel processing much more efficiently on a GPU? Um, and so, uh, that turned out to work pretty well. You had to do a couple of tricks then because you lose your ability to understand positions, so you need positional embeddings, uh, and they came up with some really cool, uh, ways to do that. And you, uh, ideally want to have the attention mechanisms sort of have multiple tries so that we can multi head attention. And that is really just what the transformer architecture is. So I would say, and maybe I'm biased because one of my best friends is, is, uh, on the original attention paper, but that was the real breakthrough. It's just like figuring out that you have this attention mechanism that actually allows you to, to, um, uh,…

AI assessment note: “one of my best friends is, is, uh, on the original attention paper, but that was the real breakthrough”

Q So as you alluded to the fundamental idea is to think in systems and not in models. Uh, so the way I understand it, uh, the general idea is to have like all components Of a rag architecture instead of a Frankenstein kind of like assembly, having all those components, uh, deeply integrated and learned together. Um, can you describe, I guess, in better terms than mine?

A Well, that was a great description, actually. I mean, so that, that's really the basic idea, right? So it's really like starting from the idea, idea that it's a system. And so, by the way, that, that also includes extraction, right? So, so there, there are lots of, Very interesting problems in terms of document understanding where existing solutions really fall short. And so if you want to have a enterprise grade rag system, you are only as good as the data that goes into that rag system. So if you can't extract the data in the right way, so if you have like a sort of table structure and it has like nested information, you can't get that out in the right way, then your rag system is going to fail. Completely. Right. So, so that is a part of garbage in, garbage out. Exactly. Yeah. So it's like beautiful data. It's like beautifully formatted PDFs, very easy to read for humans. And then, um, the AI can't do anything with that information because you can't get it out.

AI assessment note: “Well, that was a great description, actually. I mean, so that, that's really the basic idea”

Q double click on this, uh, who decides that the PDF goes into the vector database in the first place? Is that a complicated sort of like enterprise discussion? And I guess the parallel to that is that, um, is there a world where at some point the Rack system can just go fetch the information wherever it is versus having one database, uh, that's You know, consider the Rack database.

A Yeah, um, so, so that world is already here, I think, so there are different ways to do that. One is to make sure that you have access to all of these different systems, and then you synchronize that with your vector database. Or the other is to just have your language model, uh, reason and use tools and actually call into other APIs, right? So I don't have to index all of Slack if I can call Slack's search API and get the information that I need. Right. So that has benefits because then I also don't have to worry about sort of entitlements or role-based access control or things like that. So there are different strategies you can follow there. Um, but, but I think you're right. Like for enterprises that often really is just a human decision. It's like what data goes into these platforms and how do I control that data? Uh, that's, that's a really important problem. Uh, so Uh, just giving the system access to all data in a company is often not, not really the right way to do it. But ideally, um, you, you should be able to put any data that you want into it and then expect it to work. And then actually is often not true, right? So real world data is very noisy. Um, you can build a very awesome demo on a couple of PDFs and things will probably work. But then you have to scale it up to a million PDFs, and then everything breaks down. And the reason for that is that a lot of these k…

AI assessment note: “that world is already here, I think, so there are different ways to do that.”

Q that there was, uh, Claude Sonnet, 3.7, and then grog three, and then, you know, going all the way back to, to, uh, January, uh, of, of this year, deep research operator, all those things. So I'm curious, maybe it was your, uh, AI researcher hat on, uh, what do you make of all of this? What catches your imagination? What do you think is, is more or less interesting?

A Yeah, it's, uh, exciting times, right? There, there's so much happening. It's hard to keep up, but, um, Yeah, I think the new model releases are interesting. I think some people are maybe a little bit underwhelmed in terms of, you know, expectations are also getting maybe a little bit inflated with all of these releases. For me, the most exciting thing by far is DeepSeek, where that really, I think, changed the narrative in the AI ecosystem around what's possible, and who actually is an incumbent, and what is the moat that some of these companies have. Um, so, um, yeah, I, I think it's really great for, for the world that, that we have, uh, kind of an existence proof now that it's actually not that hard to do this, um, and so you don't need to invest all that much in, in data, and you can use synthetic data and, and get a pretty good model out of that, so that, that's, that's really exciting.

AI assessment note: “For me, the most exciting thing by far is DeepSeek”

Q Is that a human challenge or is that a technical challenge or both? And by human, I mean, uh, you know, internally the enterprise deciding what information goes into the, uh, Uh, into the vector database.

A Yeah, so it shouldn't be a human problem. Sometimes it is now for some companies, but it shouldn't be, right? So you should just have an AI system that can just read a document. Like, that's not, not that much to ask of an AI system, right? So, um, but, but, uh, understanding PDFs is a great example, right? How humans understand PDFs is by, like, looking at it visually, but how a machine understands it is by, Like looking at the actual like encoding, right? So the raw information that makes up that PDF. Uh, and so what we're doing is we're actually looking at the PDF the same way that a human does. So we're looking at the entire layout. We have a layout segmentation model that says, Hey, this is a chart. This is a table. This is a piece of text. And if it's a table, then it's extracted differently using a different, uh, table extraction model. And if it's a graph, then it's extracted differently because Yeah, that's a different modality for the data, right? So all of that information is then put together in, in your extraction output, and that is what ends up in your, your retrieval database, uh, for your React system.

AI assessment note: “it shouldn't be a human problem. Sometimes it is now for some companies”

Q cool stuff on Twitter, a lot of cool demos, but it feels like you're exactly at the spot where sort of the rubber meets the road in terms of actually deploying AI for real-world cases in the enterprise. What have you seen Uh, in terms of, uh, where we are in adopt, in terms of adoption and opportunities and obstacles, and where do you think we are in the cycle?

A Yeah. Um, I, I, I think we're still pretty early. There, there has been this sort of first wave where everybody wanted to do everything themselves. Um, and so, uh, we have had lots of sort of build versus buy hurdles to, to overcome there. Uh, I think people are starting to realize much more that you cannot build, A, a sort of rag platform like ours yourself, and then, then keep maintaining that forever and sort of keep up with all the latest innovations and trends that happen continuously. You see how, how quickly things change in AI, right? So, uh, I think that the market is sort of waking up to that observation much more. Um, and, and, um, in terms of maturity, I think a lot of folks initially, um, were, were thinking, Uh, about it as sort of like we need to get something in production, right? So it was kind of aiming too low. It's like, oh, we, we do like some internal enterprise search, and now we can ask like who our for one K provider is like that. That's great. You know, you could probably already do that before gen AI. Um, uh, but, um, uh, you know, that's, that's not where the ROI comes from. So that, that's why we're so focused on these really high value, Uh, knowledge worker use cases where, where you have knowledge professionals. If you can make them even 10% better at their job, you can, can save companies millions or hundreds of millions of dollars, uh, by, by do…

AI assessment note: “I think we're still pretty early. There, there has been this sort of first wave”

Q others said that, and the entire world freaked out for like two to three days. And, uh, fast forward to today, it almost It feels like some of it, uh, at least in the press, was swept under the rug, and we're just proceeding as originally planned, but it's still a big deal, so maybe, uh, explain why it's a big deal, and, and where do you see the impact?

A Yeah, I, I think why it's a big deal is, is the, the reason for that is very different from what you would see in the press, right? So, so, um, I don't know why journalists do this, but they really like Uh, sort of, um, uh, very clear adversarial story. It's like the US versus China, right? Or so something like that is very easy to understand for people. And I guess that's what people enjoy reading or sort of what people click on. And that's, that's, uh, become a part of the dominant narrative and like the Sputnik moment, like calling it that also kind of feels in like, you know, goes in that direction of like a cold war on AI or things like that. And I think that's all completely overblown. Um, but I, I think it, it, it is very interesting technology. And so what I was saying earlier, right? It's really an existence proof. It's, it's like when you have a bunch of GPUs and you know how to train a language model, uh, and, and if you can train up a pretty good base model, so that's their deep seek V three, then you can give that reasoning capabilities relatively cheaply. And, and by that, I don't mean. In, in terms of compute, I mean, in terms of data, right? So the, the data that they got, uh, came from, from, uh, uh, other language models, right? So it's synthetic data. They also had some human annotators. They're a little bit unclear about where the data really came from. Uh, …

AI assessment note: “why it's a big deal is, is the, the reason for that is very different”

Q You have a general RAG agent, and then you have specialized RAG agents. Is that right? So why is that, and who does what?

A Yeah, so the starting point is really, you can create your own agent in minutes. It's very easy to do that. You don't have to do much, and out of the box, that will already be much better than what you could have built yourself, probably with a kind of open source RAG framework. And then you can specialize that using machine learning, so you can use our Toon API. To really make all the components optimized for the specific problem that you're, you're trying to solve. Uh, so that, that's how you get to specialized reg agents where you can really hit the production bar and, and actually, uh, deploy this in, in a setting where you have much more control over what is right and wrong, right? You don't, you don't just have prompting. You can actually like train that entire system to be good at what you needed to be good at.

AI assessment note: “how you get to specialized reg agents where you can really hit the production bar”

Q the discussion, uh, you know, we're going to spend a bunch of time talking about RAG and RAG doododo, uh, and obviously a key, uh, objective of RAG is to reduce hallucination and, and all those good things, uh, at the model level. So if you take the Anthropics and the OpenAI models, uh, are you seeing progress Around, uh, hallucination control specifically, and if so, what are the drivers?

A So, on the model specifically, I, I think models still hallucinate, and I, I don't even necessarily think that that's a bad thing. Um, so, so first of all, hallucination is very ill-defined, right? I think a lot of people are kind of conflating it with being wrong. Uh, but I think hallucination is a very specific type of being wrong, where you, like, make up information that Uh, is, is not grounded in what you consider to be the ground truth, right? So, so I think for a general purpose language model, um, uh, if you deploy this in a marketing department or creative writing, then hallucination is a feature. It's not a bug, right? You want it to generate beautiful prose. Maybe you don't care that much about the factuality of things. So, so it's really a problem. The underlying problem is that we want these language models to be good at everything. So they have to be generalist, and so they also have to be useful for creative writing. Um, and, and I think, uh, that's wrong. Where, where we're headed is that we will have more specialized language models. So we have our grounded language model that has specifically been trained to be grounded, and that model hallucinates much less. So it's really much more strongly coupled to the context. Um, and, and so it's not great for creative writing, but it's very, very good at rag problems.

AI assessment note: “Where we're headed is that we will have more specialized language models.”

Q Let's talk about your, some of your, uh, fine tuning and alignment techniques, uh, Grid, KTO, Lens, what are those?

A Yeah, so, so, um, uh, KTO, um, is a different way, uh, to do, uh, DPO, so direct preference optimization. Um, so, so ROHF, when that came out, right, reinforcement learning from human feedback, everybody was like, oh, we have these kind of preferences, so essentially we have maybe two different possible answers that the language model might give, and now we want to train it to say, like, actually, this one is more preferred by humans than this one. And so this was originally kind of the secret sauce behind ChatGPT. That's why ChatGPT suddenly really started working because it captured people's preferences. But the problem with ROHF is that you need to have this very heavy reward model And that needs to be trained up on these pairwise preferences, or you need to have multiple, uh, of these generations, which is, is really problematic. So DPO then said, okay, actually we can do some smart math and then we don't need the reward model anymore, which was great. You can directly optimize on the preferences. But then when we were thinking about this, we were like, that's still not ideal because in the real world, you don't want to, uh, collect a thumbs up for every thumbs down, right? Like when, when you generate an example and somebody tells you, actually, that wasn't a good example, then you want to be able to learn from that without being told what the right example was or vice ver…

AI assessment note: “with KTO, we were like, can we directly optimize on the feedback without there being preference pairs”

Q And, uh, you alluded to this, but like, how does it even work in a research organization like FAIR? How does one decide to focus on this topic or that topic and get approval? And how does that actually work?

A No, there, there are no approvals, but now there are approvals. At the time, this, this was, this was the beautiful era of AI. I think I'm incredibly lucky to have been a part of that. Um, uh, there were no rules. Um, I, I, I arrived as a postdoc, and I think I had, like, six interns in my first year, and there was, like, I could work on whatever I wanted. Uh, I was looking at, like, Wittgensteinian, like, language games, and all kinds of weird stuff, emergent communication, like, multi-agent systems, and things that, a lot of the things that I was looking at, at the time, they're starting to, sort of, become cool now. Uh, but at the time, a lot of people thought that I was kind of crazy for even looking at these things, and, and, Uh, yeah, so I was, I was very blessed that they let me look at that stuff. Um, and so reg is really, um, really an example of that. I just like just thinking of kind of Frontier ideas. And then it turns out that they actually work. And so when the rag paper came out, I don't think a lot of people were super excited about it. It was like, oh, this is kind of cute. It happened much later. Like you can even see this in the citation profile on Google Scholar. It's like, oh, the rag paper is nice. And then it's like, holy shit, rag works.

AI assessment note: “No, there, there are no approvals, but now there are approvals.”

Q So, so what's your take then on the sort of geopolitical war, or maybe like the rough between China and the US? Is that, uh, is that overblown as well? Is that more of a media creation? How do you, what's your take on it?

A I think it's probably good for the world. I think, uh, the, the, the Chinese economy is very good at producing things, uh, uh, cheaply, uh, and in a highly optimized way. And so if you're not in the foundation model race, uh, then it could be a very good thing. So for, for me, uh, it's great if language models get commoditized, um, because then we can use them for all kinds of interesting applications, uh, in, in a much, uh, Much better way. So what we do is we contextualize the language models so that they can do their job, right? So I think that's actually where, where all the really interesting problems are right now. It's not even really about language models anymore. That has almost been solved, right? That's kind of why you see things plateauing off a little bit as well. Um, what, what's really interesting is how you make those language models do valuable things for enterprises, or how do you solve real problems? How do you deliver ROI? There, there's a lot of anticipation around, um, yeah, we need to show that, that there is a return on investment for all of these huge investments that have gone into AI. So the way to do that is to really build, build systems that solve the problems and the model, the language model is really a very small part of that much bigger system.

AI assessment note: “what we do is we contextualize the language models so that they can do their job”