Liberty: RAG Levels the Playing Field Across Different Foundation Models
Edo Liberty · No Priors Ep. 52 | With Pinecone CEO Edo Liberty · Feb 22, 2024 · at 5:27
Pinecone CEO Edo Liberty shares findings from benchmarking multiple models (OpenAI, Anthropic, Cohere, open source) with RAG over Common Crawl data.
0:00 / 0:16exact quote · 16.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Interestingly enough, many of them actually start behaving quite similarly in terms of level of accuracy, even though without RAG, they actually have quite different behaviors. So it's sort of both like a uniform improvement and a little bit of leveling the playing field.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Edo Liberty
Liberty: OSS Vector Database Competitors Are Already Struggling With Commercialization
“And in fact, we already see, even though new players in the vector database space that, that, that basically started to try to take us down, all took the open source angle. We already see them, even young as they might be, they are already struggling, struggli…”
Liberty: Foundation Models Fundamentally Flawed by Combining Reasoning With Knowledge
“Foundational models get it fundamentally wrong. When we learn how to Build the subsystems of AI correctly, and for each one of them to do their roles optimally. Either we're going to do, be able to do the, to achieve the same tasks much cheaper, faster, better…”
Liberty: RAG Over Internet Data Reduces LLM Hallucinations by 50%
“And you could see that if you augment all of them with RAG on, even on the internet, which is data that they were trained on, you can reduce hallucinations significantly up to 50% sometimes.”
Liberty: Retrofitted Vector Indexes Like pgvector Fail at Production Scale
“Those other products don't work. They don't work either because they don't scale in terms of the efficiency scale, cost, the trade-offs that they can offer, because they're not designed to do this. They're designed to do something else. They kind of thought ab…”
Liberty: Keyword search is a deeply flawed retrieval method for AI
“With other search technologies, this is again, this is the wrong search mode. If you're searching with keywords and just not finding The relevant information, because the embeddings, the contextual space in which these pieces of text, documents, or images live…”