Retrieval Augmented Generation

“We actually get a lot of credit for being the first enterprise AI company in the world. The first ones to actually bring RAG into the enterprise. The first ones to build conceptual semantic search.”

“Maybe it changes in a way that, you know, like it doesn't need to trigger RAG for like everything, but I'm pretty sure that they're going to be some tail of the distribution that we're going to do RAG still for it.”

“Very early on, we were putting lots of facts into a vector database and doing embeddings and pulling them back out using, you know, reverse look of embeddings. Rag that actually worked, but turned out to be much more complexity than was actually required. So, …”

“And I got GPD three to do in context learning, few short learning, and you know, it was kind of the First, at least to me, it was the first known implementation of RAG, Retrieval Augmented Generation, which I used to solve this problem of querying, getting GPT…”

“Just a standard LLM doing rag onto these knowledge sources and trying to answer questions is often you have conflicting answers from two different sources because that's the reality of enterprise building up knowledge bases across You know, a huge company.”

“Like, you know, before, if you imagine like a year or two ago, you had to put a lot of effort into very bespoke ragstacks and routing, but now LLMs are getting more capable of doing more agentic behavior. As a result, you have to change your architecture. Othe…”

“Agentic RAG is just generally better than RAG. Even that initial incremental step of making that doing query reformulation, so when you receive that initial query, being able to break it down into sub-queries so that you can better match those queries to docum…”

“I think there were a lot of advances in RAG, and the biggest beneficiaries of these advances were the application companies for whom, you know, retrieval was a critical unlock.”

“I think especially where, where things are headed, like with more complex rags and agents and so forth, you're going to have to have some type of observability or like internal tracing in order to have, to do meaningful automated red teaming.”

“Data preparations for Rack is extremely important. And I would say that's like in the, a lot of the companies that I have seen, that's like the biggest performance in their Rack solutions coming from like better data, data preparations, not agonizing over like…”

“The big move was retrieval augmented generation, where you kind of run a search. So you have a master corpus of all of the relevant tax research materials. You run an intelligent search, and you've got, like, really good vector embeddings, and you're chunking …”

“The brilliant thing about leveraging retrieval augmented generation is we can just curate the data. And if we're confident about the currency of the tax database and the content in the database, and we're confident about our retrieval algorithms, our Prompt en…”

“We're starting to see the labeling move into enterprises and into governments because you can only get so far with off the shelf plus rag, plus some fine tuning based on recorded data.”

“Most of what I'm describing is about the RAG piece, not the core model piece to influence the core model is probably extremely hard. And maybe you will see the impact a year later.”

“And I think also the LLM is, is probably not going to say your product if it didn't show up anywhere on the RAG. So I think that's where most of the interesting stuff is for, from an optimization perspective.”

“If you feed in derivatives of derivatives into the model, you will basically take the wisdom of the crowd and that will shrink and you'll have a single opinion on everything, which is really bad.”

“Memory retrieval at large scale is just retrieval, right? I kind of view them as... In the case of sophisticated memory retrieval, it is just like a, you know, complex rag system in the same way we talked about with like Varun and building windsurf. It's kind …”

“Where we find that chatbots and RAG, which is essentially Taking documents and putting them into a vector database so AI can read them, right? AI that can read your documents in a custom way is RAG. When you think about a chatbot in RAG, the value there is giv…”

“And so quad code actually doesn't use this technique called rag. Instead, what it does is it just searches files the same way that a human would.”

“Ambience internally, we use prompting, we use chaining, we'll use RAG, we use fine tuning. It will use SFT and RFT.”

“It'll grab it, and then that's all you have, and most likely, for any meaningful query it will not have given you the information that you need to actually go do the task. So rag agents are actually, are pretty weak.”

“While drag is good for casual search, you need a complex workflow under the herd that is explainable, that is auditable, that is guaranteed to be accurate and correct, is important for solving many of these enterprise problems.”

“Hebbia were actually the first to turn that into a product. So it's like a very close thing to my heart. So back in 2020, we were the first people to actually productionize it, roll it out.”

“And so RAG has obviously become very prevalent in a wide variety of industries and people use it for a lot of different things. We've spoken with different firms that they were like, okay, well, I'm just going to continue to add more and more data to the corpu…”

“If you just do like pure, like rack on like such a big base code files, it'll just be like pretty bad at a certain point, you know, on a single code base. Sure. I can see it, but Tens of thousands of code bases. It's tougher. But I think actually the wiki page…”

“Part of the initial interest for Reducto is RAG as a paradigm introduced this need for thinking about chunking, for example. So it wasn't just, can you parse the documents? It was after you parse it, what do I feed into my VectorDB? That was a, Initial selling…”

“Agents are now getting a lot more powerful. They are, You know, they're getting they're sort of shifting from sort of basic two step rack kind of application flow where you take a task, you find some information, and then you make AI work on it to generate the…”

“I think that's a more natural abstraction for connecting AI with knowledge and all the arguments for doing rag. I think the natural concept there is search.”

“I think actually that a very strong baseline is the classical BM-Five like algorithm that's been around for 30 years, right? It's keyword matching, but it offers a very useful baseline for a lot of different search use cases because it gives you that baseline,…”

“I think like today, 90 plus percent of it in enterprises is retrieval event generation, or it's, you know, using retrieval, it's sort of a chat on top of unstructured data.”

“The reality is something gets lost in the chunking process in the, okay, well, those tend to, you know, like you don't really get the whole picture, so to speak, and maybe not even the right set of dimensions on the kind of broader picture. And it makes intuit…”

“Why RAG became the way you name these things is because it's generative, right? So we were the first ones to have a generative model there.”

“Those are often the gateway to messing around with other stuff, right? Like once, once you punch a hole in the defenses with something like a jailbreak, if it's part of a larger system or architecture, like a rag or agent that would give an attacker a lot of r…”

“It's a bit dirty because, of course, you know, you are limited by the amount of data you can input, right?”

“What really pushes model sizes are the efficiency rather than specializing. So meaning that if you can do the same performance with a smaller model that is fine tuned with rag or whatever, then you'll do it with a smaller because again, less is better.”

“We also have we have retrieval mechanisms, if required. So we natively try to use the context as much as it's available beyond which you know, we have, like, a rag setup to figure out”

“The tricky thing for RAG, it really works well because a lot of these things are doing like cosine distance, like a dot product kind of a thing, and that kind of gets challenging when your query side has multiple different attributes. The dot product doesn't r…”

“Just to add to that, I think like, just like a simple rule of thumb that we use is like, if it's the most recent set of research tasks where the user is likely to ask lots of follow-up questions, that should be in context. But like, as stuff gets 10 tasks ago,…”

“I think existing kind of rag pipelines also will probably be obsoleted. The frameworks, I have not found one, like, there's no general Way to do RAG that works like it's really highly dependent on the data.”

“What we've learned is, like, doing the traditional, like, embedding and RAG is suboptimal. We kind of built our own using small models to do really massively parallel retrieval, which I think is going to be maybe more common in the future.”

“We end up building a product studio, which is the first to productionize RAG, Retrieval Augmented Generation, also in 2020.”

“That's a bit of a plot twist over here. We, I actually don't think RAG works at all.”

“At least in my experience, rag from language to code doesn't work super well.”

“We had this experience over and over again where you could have any, an embedding model of any quality, so you could have a really, really bad embedding model, or you could have a really, really good one by, and by any measure of good, and for the final RAG ap…”

“Ultimately lots of those technology can become, in isolation, pretty much commoditized. Meaning it's not hard to build RAG or agent builder.”

“Staying on the main branch, staying on the core capabilities and supplementing it with fine tuning and additional techniques like RAG seems to be the approach that's winning right now.”

“This is very different from traditional rag or Q&A or document processing for a chatbot. Like, the kinds of queries that people are, people want to use .etl for can be expressed as etl style sweep and harvest, kind of, I want to look at my entire dataset. I wo…”

“We realized just like with rag where we had invented rag already, we're doing it at a web scale, but it wasn't yet a term.”

“The next frontier after RAG is planning and reasoning.”

“If we just use GPT-IV plus RAG, what do we get? It's, like, a measly three percent. And then up to the most recent submissions where they get up to 43%.”

“My guess would be that, like, long context works, but it's sort of a lie as far as your accuracy, and that rag matters no matter what, because even in the longest context windows, you can't fit the whole code base.”

“Unambiguously, people are doing rag. So that one is, you know, it's like simple and obvious. Probably around 50% of the use cases that we see in production involve rag of some sort.”

“Some of the things that are good in the assistance API is hosted tools. People really like posted tools and especially RAG. And then some things that are, you know, less intuitive is just how many API requests you need to get going with the Assistant's API.”

“What we found in practice is that broad category of technology investment is woefully insufficient for almost any meaningful customer experience. If you think about, you know, all of the interactions you've had with brands that you care about, what percentage …”

“RAG based systems are so Are so common here, and so, and probably will remain, even with the advent of fine-tuning availability, they're going to remain a useful paradigm.”

“Something over the next 12 months people will be spending time thinking about is how to, like, where to use RAG, where to use fine-tuning, where to use KV cache storage, you know, and how to use state.”

“Today when people are running RAG or prompt engineering those are search. That's not AI. It's like keeping the AI frozen and fixed.”

“You can have prompts that are composable and that have different sort of command units that like may or may not be present based on the semantics of the content that is being populated into the rag context window. And so that, that's something we make great us…”

“Right now for implementing Rack the bottleneck seems to be that, you know, it's not very hard to implement it, right? You can just connect the components and have your Rack system ready very quickly. But the bottleneck seems to be the quality. Of the response …”

“And my long term vision here is that some of the software engineering layers on top of the networks will be less and less needed when the networks are more and more clever.”