Grokking
topic on 2 shows · 2 statements across 2 episodes
Invest Like the Best 
the MAD Podcast
2 statements about Grokking, every show
Dan Roberts: AI models do not experience discontinuous emergence or grokking
“You have these crazy, huge systems that have all sorts of interesting phenomena, and, you know, if you think about it the right way, they don't grok. There's just this nice continuity.”
Patel: Scaling model size without new data causes memorization over generalization
“And if you make the model bigger and bigger and bigger without the data changing, you just memorize everything. And actually, it starts to get worse again because it never had the opportunity to generalize because the model was so big and there's so many weigh…”