model parallelism
1 statements across 1 episodes · 0 bullish · 1 bearish · 1 people on the record · first statement Dec 7, 2024 by Sarah Chieng · across every show →
Everything said about model parallelism, oldest first
Dec 7, 2024 negative
Cerebras avoids model parallelism in production due to communication overhead
“There's a lot of communication overhead with model parallelism. You have to share activation tensors, and that is why in this paper and, you know, in production, Cerebra's focus on data parallelism. So all of this is mentioned in the paper as well, but model p…”