State Space Models
topic on 5 shows · 11 statements across 8 episodes
Latent Space 
the Neon Show 
No Priors 
the MAD Podcast 
Big Technology
11 statements about State Space Models, every show
Catanzaro: Combining SSMs and transformers produces smarter AI models than either alone
“Using both of these together was actually better than using either one on their own. And that is independent of the speed benefit. That is just the model is smarter.”
Catanzaro: Hybrid state-space transformer architectures are widely adopted in frontier AI
“It's become, I think, Quite widely adopted to use some sort of state space model in conjunction with full attention for the base architecture.”
AI capabilities would have been achieved even without inventing transformers
“I think if we hadn't invented the transformer, we would have gotten there with whatever LSTM you know, state space model, whatever, anything else people were developing, we would have gotten there.”
Kamath: State space models are not a big leap over transformers
“Our strong belief is like state space models are not like that big a leap over transformers.”
Kamath: JEPA architectures are closer to AGI than state space models
“In our opinion, like those models are actually closer to how we can build AGI than like state space models, which is just a, Play on compute capacity.”
Dan Fu: Some top audio models use state space architectures
“So some of the best audio models in the world are at least partially based on state space models.”
Cheah: Non-positional attention architectures remain stable beyond trained context
“One key advantage of this alternate attention mechanic that is not based on token position is that the model don't suddenly become crazy when you go past the eight K training context or a million context. It is actually still stable. It's still, it's able to r…”
Howard: AI developers will spend 12 months mapping RAG, fine-tuning, and KV caching
“Something over the next 12 months people will be spending time thinking about is how to, like, where to use RAG, where to use fine-tuning, where to use KV cache storage, you know, and how to use state.”
Gu: State Space Models can be applied to almost all data types
“So it really can be applied to pretty much everything. So just like kind of Transformers, these are applied to everything. So can these sort of models over the course of research over a few years, we kind of realized that there are different advantages for dif…”
Gu: Early SSMs excelled at raw signals but lagged Transformers on text
“The first types of models we were looking at were really good actually at modeling kind of these raw waveforms raw pixels, things like that, but not as good at modeling text, and transformers are way better there.”
AWS AI Chief: AI architectures beyond pure transformers are absolutely necessary
“I actually think, ah, new architectures are absolutely necessary in the future, ah, and, ah, there are already hybrid architectures evolving. I mean, ah, I mean in terms of state space models to actually connectors hybrid models between state space to transfor…”