Epoching
topic on 1 show · 1 statements across 1 episodes
1 statements about Epoching, every show
Morcos: Epoching high-quality data beats training on new average data
“Repeating higher quality tokens is almost always better than seeing net new lower quality tokens. So, like, epoching over higher quality data, almost always better than getting the same amount of new data of an unknown quality, or of average quality, average i…”