Retrieval-Augmented Generation

“I think a lot of people have thoughts about that, but, like, for what it's worth, I don't think the final state will be ragged. I think it will be some, like, fancy algorithm or architecture where you, like, bake it into, like, the architecture of the model it…”

“RAG is basically just a hack, but it turns out it's a very good hack because what is RAG? RAG is you keep the model fixed, and you just figure out a good way to, like, stuff stuff into the prompt of the language model. Everything that we're doing nowadays in t…”

“There's always going to be some curve regardless of like the performance of the best performing models of like cost versus performance. And so what RAG does is it does provide extra data points along that access because you kind of control the amount of contex…”

“Another example here is actually LLM based reasoning, like LLM based chain of thought reasoning. You can take a question, break it down into smaller components and use that to actually send to your retrieval system. And that gives you better results since kind…”

“Building, like, RAG from scratch. I mean, I think everybody should do it, I think. Like, I would check out the guide if you guys haven't already, and I think it's in our docs, but instead of just using you know, either the kind of, like the retriever query eng…”

“RAG is like such a inefficient hack, really, isn't it? It's like, You know, segment up my data in some somewhat arbitrary way, embed it, ask questions about that, you know, hope that my embedding, you know, model embeds questions in the same embedding space as…”

“I don't think RAG is enough for that kind of thing. But RAG is certainly enough for, like, user preferences and things like that.”

“Like, I think it's generally been shown that if you have the space to just put The raw files inside of a big context window. That is still better than chunking and retrieval. It just is.”

“Combining all those techniques all together, right? So we'll give you essentially another boost, right? So that kind of one thing that we learn on the technical side.”

“And like, at the end of the day, it's just all meta-learning, right? Like, all we want is, like, the best meta learning workflow or meta learning setup possible to be able to adapt the model to do anything.”

“You can have prompts that are composable and that have different sort of command units that like may or may not be present based on the semantics of the content that is being populated into the rag context window. And so that, that's something we make great us…”

“Something over the next 12 months people will be spending time thinking about is how to, like, where to use RAG, where to use fine-tuning, where to use KV cache storage, you know, and how to use state.”

“Some of the things that are good in the assistance API is hosted tools. People really like posted tools and especially RAG. And then some things that are, you know, less intuitive is just how many API requests you need to get going with the Assistant's API.”

“If we just use GPT-IV plus RAG, what do we get? It's, like, a measly three percent. And then up to the most recent submissions where they get up to 43%.”

“My guess would be that, like, long context works, but it's sort of a lie as far as your accuracy, and that rag matters no matter what, because even in the longest context windows, you can't fit the whole code base.”

“The next frontier after RAG is planning and reasoning.”

“This is very different from traditional rag or Q&A or document processing for a chatbot. Like, the kinds of queries that people are, people want to use .etl for can be expressed as etl style sweep and harvest, kind of, I want to look at my entire dataset. I wo…”

“We had this experience over and over again where you could have any, an embedding model of any quality, so you could have a really, really bad embedding model, or you could have a really, really good one by, and by any measure of good, and for the final RAG ap…”

“At least in my experience, rag from language to code doesn't work super well.”

“What we've learned is, like, doing the traditional, like, embedding and RAG is suboptimal. We kind of built our own using small models to do really massively parallel retrieval, which I think is going to be maybe more common in the future.”

“I think existing kind of rag pipelines also will probably be obsoleted. The frameworks, I have not found one, like, there's no general Way to do RAG that works like it's really highly dependent on the data.”

“We also have we have retrieval mechanisms, if required. So we natively try to use the context as much as it's available beyond which you know, we have, like, a rag setup to figure out”

“Just to add to that, I think like, just like a simple rule of thumb that we use is like, if it's the most recent set of research tasks where the user is likely to ask lots of follow-up questions, that should be in context. But like, as stuff gets 10 tasks ago,…”

“The tricky thing for RAG, it really works well because a lot of these things are doing like cosine distance, like a dot product kind of a thing, and that kind of gets challenging when your query side has multiple different attributes. The dot product doesn't r…”

“The reality is something gets lost in the chunking process in the, okay, well, those tend to, you know, like you don't really get the whole picture, so to speak, and maybe not even the right set of dimensions on the kind of broader picture. And it makes intuit…”

“I think actually that a very strong baseline is the classical BM-Five like algorithm that's been around for 30 years, right? It's keyword matching, but it offers a very useful baseline for a lot of different search use cases because it gives you that baseline,…”

“I think that's a more natural abstraction for connecting AI with knowledge and all the arguments for doing rag. I think the natural concept there is search.”

“If you just do like pure, like rack on like such a big base code files, it'll just be like pretty bad at a certain point, you know, on a single code base. Sure. I can see it, but Tens of thousands of code bases. It's tougher. But I think actually the wiki page…”

“Ambience internally, we use prompting, we use chaining, we'll use RAG, we use fine tuning. It will use SFT and RFT.”

“Memory retrieval at large scale is just retrieval, right? I kind of view them as... In the case of sophisticated memory retrieval, it is just like a, you know, complex rag system in the same way we talked about with like Varun and building windsurf. It's kind …”

“I think especially where, where things are headed, like with more complex rags and agents and so forth, you're going to have to have some type of observability or like internal tracing in order to have, to do meaningful automated red teaming.”

“I think there were a lot of advances in RAG, and the biggest beneficiaries of these advances were the application companies for whom, you know, retrieval was a critical unlock.”

“Agentic RAG is just generally better than RAG. Even that initial incremental step of making that doing query reformulation, so when you receive that initial query, being able to break it down into sub-queries so that you can better match those queries to docum…”

“Very early on, we were putting lots of facts into a vector database and doing embeddings and pulling them back out using, you know, reverse look of embeddings. Rag that actually worked, but turned out to be much more complexity than was actually required. So, …”