Ramachandran: Needle-in-a-haystack retrieval benchmarks fail to model real code retrieval
Anshul Ramachandran · Windsurf: The Enterprise AI IDE · Dec 13, 2024 · at 13:45
Anshul Ramachandran, VP of Product at Codeium, explains why AI retrieval for code requires pulling distributed context rather than finding single discrete facts.
0:00 / 0:18exact quote · 18.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“A lot of the benchmarks for these embedding-based systems are like needle in the haystack problems. Like, I want to find this one particular piece of information out of all this potential context. That's not That's not really what actually is necessary for doing software engineering because code is a super distributed knowledge store. You actually want to pull in snippets from a lot of different parts of the code base in order to do the work”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Anshul Ramachandran
Ramachandran: SWE-bench and HumanEval do not reflect real professional software engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have SweeBench, that's cool, no actual Professional work looks like Sweebench, like human eval, same thing.”
Ramachandran: SWE-bench and HumanEval do not reflect real professional engineering
“Most evals and benchmarks that exist out there for software development is kind of bogus. There's not really a better way of putting it. Like, okay, you have sweet bench. That's cool. No actual Professional work looks like Sweebench. Like, human eval, same thi…”
Ramachandran: AI flows will succeed AI agents in developer tooling
“Copilots are collaborative, but can only work on small scopes. Agents can do larger tasks, but are not collaborative. Both are very useful, but we realize that this real magic will happen when the AI has the ability to be both collaborative and independently p…”
Ramachandran: Cascade goes further than any other agentic system
“This allows Cascade to be independent, but Cascade takes it further than any other agentic system. By also generating commands to be run.”
Ramachandran: Tracking developer trajectory allows AI to infer intent without prompting
“If we're actually being able to understand the kind of the trajectory of what developers are doing within the editor, right, if we actually are being able to see, like, oh, the developer just went and opened up this part of the directory and tried to view it, …”
Ramachandran: Capturing Editor Trajectories Lets AI Infer Intent Without Explicit Prompting
“If we actually understand that trajectory, then our ability for the AI to just be immediately be like, Oh, I understand your intent. This is what you want to do without you having to spell it all out for it. That is one like that kind of like magic would reall…”