Huang: Needle in a Haystack is a primitive LLM benchmark prerequisite
Mark Huang · How to train a Million Context LLM — with Mark Huang of Gradient.ai · May 31, 2024 · at 42:55
Mark Huang (co-founder of Gradient.ai) discusses long-context LLM evaluation metrics on the Latent Space podcast with Swix.
0:00 / 0:23exact quote · 23.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“I think needle in a haystack is definitely, like, the standard for presenting the work in a way that people can understand and also proving out. I would say, like, I view it as, like, a primitive. That you have to pass in order to give the model any shot of doing something that combines both like a more holistic language understanding and like instruction following, right?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mark Huang
Huang: True AI agents require measurable probability improvements per node
“It's like on each stage of the node, you're gonna have to see a marginal improvement in the probability of success for that particular workload because of non-determinism.”
Huang: Google's internal AI tooling was far superior to competitors
“Google was using AI for systems before everybody else too, right? They invented a transformer, and their internal set of tooling was just so far superior to everything else. Like, it's really hard for people to go back after seeing that.”
Huang: RAG versus fine-tuning is fundamentally just meta-learning
“And like, at the end of the day, it's just all meta-learning, right? Like, all we want is, like, the best meta learning workflow or meta learning setup possible to be able to adapt the model to do anything.”
Huang: PoSE breaks down on needle-in-a-haystack at 500k tokens
“It does start to break down a little bit more on the longer, longer context. So, like, 500,000 to a million it appeared that it doesn't hold as well specifically for, like, needle in the haystack.”
Huang: Adding one billion tokens cannot teach trillion-token models new knowledge
“All models these days are now double-digit trillions, right? So it's kind of a drop in the bucket if you really think I can just put, you know, a billion tokens in there, and I actually think that the model's gonna truly learn new Information.”