Data Parallelism
topic on 1 show · 1 statements across 1 episodes
1 statements about Data Parallelism, every show
Cerebras avoids model parallelism in production due to communication overhead
“There's a lot of communication overhead with model parallelism. You have to share activation tensors, and that is why in this paper and, you know, in production, Cerebra's focus on data parallelism. So all of this is mentioned in the paper as well, but model p…”