Liu: Claude 3 Haiku outperforms OpenAI models at function calling
Jason Liu · High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor · Apr 24, 2024 · at 21:57
Jason Liu (creator of Instructor) evaluates Anthropic's Claude 3 models against OpenAI models based on production workloads.
0:00 / 0:09exact quote · 9.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Overall, I'm like super happy with the anthropic models compared to the OpenAM models. Like, Sonnet is very cost effective. Haiku is, in function calling, it's actually better.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jason Liu
Liu: LLM observability startups ignore full systems; just use Postgres
“The issue really is the fact that these observability companies isn't actually doing observability for the system, it's just doing the LLM thing. Like I still end up using like Datadog, right? Or like, you know, Sentry to do, like, latency. And so I just have …”
Liu: Companies abandon LLM frameworks to regain control over prompts
“So much of it is changing that if you give control of these systems away too early, you end up ultimately wanting them back. Like many companies I know that I reach out or ones were like, oh, we're going off of the frameworks because now that we know what the …”
Liu: App-layer AI startups should avoid hiring traditional machine learning engineers
“I think a lot of these app layer startups should not be hiring MLEs because they end up churning.”
Liu: XGBoost rankers will beat LLMs for large-scale tool selection
“Yeah, my money is on the rankers because you can do those so easily, right? You could just say, well, given the embeddings of my search query and the embeddings of the description, I can just train XGBoost and just make sure that I have very high, like, MRR, w…”
Liu: AI agents should use explicit DAG workflows over ReAct loops
“Instead of doing like a react type reasoning loop, I think my belief is that we should be using like workflows, right? If we do this, then we always have a request and a complete workflow. We can fine tune a model that has a better workflow. Whereas it's hard …”
Liu: LangChain and LlamaIndex can hit $100M revenue, but billions uncertain
“I think the bigger challenge is like, okay, a hundred million dollars, probably pretty, pretty easy. It's just time and effort. And they have both like the manpower and the money to sort of solve those problems. I think it's just like, again, if you go the VC …”