The Pile
3 statements across 3 episodes · 1 bullish · 0 bearish · 3 people on the record · first statement Dec 17, 2023 by Beyang Liu · said 14 times in 7 episodes since 2023 · across every show →
Mentions by year
brought up most by Eugene Cheah (7), Shawn Wang (2), Ce Zhang (1), Beyang Liu (1)
tap a year for its mentions
2025 3 mentions in 2 episodes 2 per episode
2024 2 mentions in 2 episodes 1 per episode
Everything said about The Pile, oldest first
Dec 17, 2023 bullish
Liu: Synthetic data and task-specific fine-tuning provide alpha for code automation
“I feel like most models today, they still use, like, combination of, like, the stack and the pile as, like their training corpus but you can only stretch that so far. At some point, we need more data and I don't know. I think there's still more alpha in, like,…”
Jul 28, 2025 neutral
Mohan: Codeium scraped public code because The Pile was too small
“We ended up basically using a lot of open, I guess, permissively licensed code in the public internet, mainly because I think also the pile is, is fairly a small subset. I think maybe after we started, there was the stock that was also came to be, but for us, …”
Sep 23, 2025 neutral
Bachman: 90% of documents in web pre-training datasets are under 2k context
“If you take like any old, like the pile or one of the more modern ones, basically any common crawl derived, scrape, open web text, any of these things, you're going to find that almost all documents are very short. There's some long context documents, but the …”