The Pile, every mention
9 scenes · ← back to The Pile
tap a year for its mentions
every year anyone Eugene Cheah 7Shawn Wang 2Ce Zhang 1Beyang Liu 1
Verbatim, from the transcripts: the passages where The Pile comes up
⚡️ Beyond Transformers with Power Retention
- ▶ 30:30 unnamed speaker If you take, uh, like any old, like the pile or one of the more modern ones, basically any common crawl derived, scrape, open web text, any of these things, you're going to find that almost all documents are very short.
🕰️ The Oral History of Windsurf (ft. Varun Mohan, Scott Wu, Jeff Wang, Kevin Hou, Anshul R)
- ▶ 12:58 Shawn Wang I actually assumed that you would have grabbed your data from the pile, which is the Luther AI open source code information, but apparently you scraped your own stuff. 2 times in the scene
How AI is Eating Finance - with Mike Conover of Brightwave
- ▶ 1:02:55 unnamed speaker Like, I mean, the pile, the pile is still around and like core to most of these training runs versus all the models people trained a year ago.
Building an open AI company - with Ce and Vipul of Together AI
The "Normsky" architecture for AI coding agents — with Beyang Liu + Steve Yegge of SourceGraph
- ▶ 1:11:39 Beyang Liu It's just, like, I feel like most models today, they still use, like, combination of, like, the stack and the pile, uh, as, like, uh, their, their training corpus, um, but you can only stretch that so far.
RWKV: Reinventing RNNs for the Transformer Era
- ▶ 39:04 Eugene Cheah Is RWKV for the power and Raven. 5 times in the scene
- ▶ 42:35 Eugene Cheah Power Plus, Red Pijama, then subsequently, some of the code, code data.
- ▶ 1:10:07 Eugene Cheah It's, it's the PAL.
FlashAttention-2: Making Transformers 800% faster AND exact
- ▶ 59:02 unnamed speaker And like the PTA models are great, but like the pile and like the stack, like, you know, everybody uses them, you know?