Attention Layer
topic on 1 show · 1 statements across 1 episodes
1 statements about Attention Layer, every show
The 'Original Sin' of Transformers Is Pairing Memory-Bound and Compute-Bound Layers
“And I would say the original sin of Transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and …”