Internet Data
topic on 5 shows · 8 statements across 8 episodes
Lenny's Podcast 
No Priors 
Sourcery 
the a16z Podcast 
20VC
8 statements about Internet Data, every show
Das: Arbitrary LLM scaling fails once internet training data is exhausted
“It turns out that the scaling laws are proportional to the amount of data you can train on. So you can't just arbitrarily make the model super big if the data set is just the internet. Yeah, maybe you can accentuate that with other kinds of like proprietary in…”
Huyen: Internet data is maxed out, making post-training the key AI differentiator
“At some point, we are actually, like, have kind of maxed out on, like, internet data, right? And then people, like, text data, people max out. I think a lot of people are doing, like, with other data, like audios and videos, and, like, everyone's trying to thi…”
Gil: Physical-world data bottlenecks explain why AI breakthroughs began in language models
“Anytime you go into the physical world, it's always harder to generate data, and that's one of the reasons that the language models where you just effectively collect the wisdom of the internet digitally are the first places where we've really seen this scale …”
Wang: AI model training has exhausted all available internet data
“At a super super high level I think we've used up all the easy data. We've used up all of the internet data.”
Alexandr Wang: The AI industry has exhausted all easy internet training data.
“And we've sort of, as a community, we have, we've had easy data, which is all the data on the internet and we've kind of exhausted all the easy data, and now it's about, you know, forward data production that has high supervisory signal that is basically very …”
Doshi: Vision AI can access infinite real-world data unlike internet text
“Whereas like with vision, at least you can like make a robot that just like travels down the street and like just keeps taking pictures of everything. You can get like infinite training data with vision, but it might be trickier to like sort of filter and clea…”
Mostaque: AI developers must stop pre-training models on uncurated internet data
“We've got to stop pre-training big models on all the crazy crap of the internet. And like, we've got to do it fast because this is coming like a train.”