Internet Text
topic on 2 shows · 2 statements across 2 episodes
2 statements about Internet Text, every show
Bachman: Compute-optimal models on internet text don't need long context
“In general, most internet text has mostly short-term structure. There's just not that much value in capturing long-term structure, and so compute optimal models on internet text actually don't have that long context, and so, of course, you're perfectly fine us…”