Liu: XGBoost rankers will beat LLMs for large-scale tool selection
Jason Liu · High Agency Pydantic over VC Backed Frameworks — with Jason Liu of Instructor · Apr 24, 2024 · at 19:54
Jason Liu, creator of Instructor, argues that dedicated rankers will solve tool selection for large catalogs rather than frontier model context scaling.
0:00 / 0:27exact quote · 27.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Yeah, my money is on the rankers because you can do those so easily, right? You could just say, well, given the embeddings of my search query and the embeddings of the description, I can just train XGBoost and just make sure that I have very high, like, MRR, which is like mean reciprocal rank. And so, like, the only objective is to make sure that the tools you use are in the top and filter. Like, that feels super straightforward, and you don't have to actually figure out how to fine tune a language model to do tool selection anymore.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Jason Liu
Liu: LLM observability startups ignore full systems; just use Postgres
“The issue really is the fact that these observability companies isn't actually doing observability for the system, it's just doing the LLM thing. Like I still end up using like Datadog, right? Or like, you know, Sentry to do, like, latency. And so I just have …”
Liu: Companies abandon LLM frameworks to regain control over prompts
“So much of it is changing that if you give control of these systems away too early, you end up ultimately wanting them back. Like many companies I know that I reach out or ones were like, oh, we're going off of the frameworks because now that we know what the …”
Liu: App-layer AI startups should avoid hiring traditional machine learning engineers
“I think a lot of these app layer startups should not be hiring MLEs because they end up churning.”
Liu: AI agents should use explicit DAG workflows over ReAct loops
“Instead of doing like a react type reasoning loop, I think my belief is that we should be using like workflows, right? If we do this, then we always have a request and a complete workflow. We can fine tune a model that has a better workflow. Whereas it's hard …”
Liu: LangChain and LlamaIndex can hit $100M revenue, but billions uncertain
“I think the bigger challenge is like, okay, a hundred million dollars, probably pretty, pretty easy. It's just time and effort. And they have both like the manpower and the money to sort of solve those problems. I think it's just like, again, if you go the VC …”
Liu: Fine-tuning on proprietary data at scale always beats off-the-shelf models
“We, because, I mean, at this point, we would have, like, you know, three million pieces of inventory, over, like, a billion interactions between users and clothes. Any kind of fine-taining would definitely outperform like, some off-the-shove model.”