SSM layers
1 statements across 1 episodes · 1 bullish · 0 bearish · 1 people on the record · first statement Jun 27, 2024 by Albert Gu · across every show →
Everything said about SSM layers, oldest first
Jun 27, 2024 bullish
Gu: Optimal hybrid models use a 10:1 ratio of SSM to attention
“People have found that the optimal ratio tends to be mostly SSM layers with a little bit of attention. So maybe a ratio of like 10 to one, I know of at least Probably like five groups that have independently verified that this is kind of the optimal ratio of t…”