open web text
1 statements across 1 episodes · 0 bullish · 0 bearish · 1 people on the record · first statement Sep 23, 2025 by Diego Bachman · across every show →
Everything said about open web text, oldest first
Sep 23, 2025 neutral
Bachman: 90% of documents in web pre-training datasets are under 2k context
“If you take like any old, like the pile or one of the more modern ones, basically any common crawl derived, scrape, open web text, any of these things, you're going to find that almost all documents are very short. There's some long context documents, but the …”