pre-training data

2 statements across 2 episodes · 0 bullish · 1 bearish · 2 people on the record · first statement May 31, 2024 by Mark Huang · across every show →

Everything said about pre-training data, oldest first

May 31, 2024 bearish

Insight

Huang: Adding one billion tokens cannot teach trillion-token models new knowledge

“All models these days are now double-digit trillions, right? So it's kind of a drop in the bucket if you really think I can just put, you know, a billion tokens in there, and I actually think that the model's gonna truly learn new Information.”

Mark Huang May 31, 2024 ▶ 31:49 How to train a Million Context LLM — with Mark Huang of Gradient.ai

Jan 23, 2026

Assertion Not checkable as stated

Yi Tay: AI Labs Prioritize Data Efficiency Because the World Lacks Tokens

“I think in general, the, like learning more, like extracting more from varied data points is definitely valuable, but I think that's what related to the fact that we're like running out of Tokens in the world.”

Yi Tay Jan 23, 2026 ▶ 57:07 Captaining IMO Gold, Deep Think, On-Policy RL, Feeling the AGI in Singapore — Yi Tay

Turn any episode into a week of clips.

This entire site, over 200 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

pre-training data

Everything said about pre-training data, oldest first

Turn any episode into a week of clips.