SSM Layers
topic on 1 show · 1 statements across 1 episodes
1 statements about SSM Layers, every show
Gu: Optimal hybrid models use a 10:1 ratio of SSM to attention
“People have found that the optimal ratio tends to be mostly SSM layers with a little bit of attention. So maybe a ratio of like 10 to one, I know of at least Probably like five groups that have independently verified that this is kind of the optimal ratio of t…”