Chase: Open-source models still lag behind Claude 3 and GPT-4
Harrison Chase · No Priors Ep. 57 | With LangChain CEO and Co-Founder Harrison Chase · Mar 28, 2024 · at 22:06
Harrison Chase, CEO of LangChain, shares his view on how open-source LLMs compare to proprietary frontier models from Anthropic and OpenAI for agent workflows.
0:00 / 0:47exact quote · 47.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Like there's, I think we see increasingly interest in open source, but the reasoning abilities are still just like lagging behind Cloud three or GPT four. And I think like for a lot of the applications that it kind of, it probably depends on the types of applications that you're building. But a lot of the applications that LinkedIn is focused on with this kind of like reasoning aspect, those are just so crucial. And I don't think we see Super compelling. I don't, I still don't think we see super compelling reasoning abilities in the open source models. And maybe that's one of my hot takes, but I think for a lot of the LinkedIn apps, the open source models maybe don't live up to a lot of kind of like the Twitter hype or Twitter excitement, at least not yet.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Harrison Chase
Chase: Long context windows will not replace chaining and AI agents
“There are also things where it requires iterations. You need to like decide what to do, interact with the environment, get that back. So this whole idea of chaining and agents, I don't like,
That's less around context windows and more around interacting with t…”
Chase: Needle-in-a-haystack benchmarks fail to represent real RAG reasoning
“That, that actually really doesn't reflect a lot of RAG use cases in, in my opinion, because like that's the needle in the haystack is like, okay, given this long context, can I find a single information point? But oftentimes RAG is about seeing multiple infor…”
Chase: AI agent memory is extremely nascent and lacks interesting developments
“I feel it's like a field that's just, like, super, super nascent. Like, I don't, I actually am underwhelmed at the amount of, like, really interesting stuff that's going on there.”
Chase: Successful production AI agents operate as controlled state machines
“And I think the things that we see making it into production and informed a lot of the development of laying graph is, or is something in the middle where it's like this controlled state machine type thing.”