Dec 24, 2024 · 42m · latent-space
2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
At NeurIPS 2024, Dan and Eugene examine the emergence of post-transformer architectures, detailing how State Space Models (SSMs) and RWKV overcome quadratic attention bottlenecks through subquadratic scaling, hardware-model co-design, and hybrid systems.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 0.7 Guest teaching 0.5 Guest disagreement 0.5 The hosts pushing back 0.5
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 37:49 Eugene polling the audience on real 2M context usage
Eugene playfully challenges the practical utility of extreme context windows by polling the audience on whether anyone actually uses multi-million token prompts.
Hardest push from the hosts ▶ 36:48 Host challenges practical necessity of massive context scalingThe host directly questions the premise of extreme context benchmarks, asking why anyone would submit a two-million token query when RAG exists.
Biggest teaching moment ▶ 38:35 Eugene explains the VRAM backpropagation bottleneck during trainingEugene educates the room on the reality of training recurrent architectures on long contexts, explaining that state unrolling still consumes massive VRAM during backpropagation.
The host holds their own ▶ 38:08 Host asserts frequent real-world use of massive context windowsWhen Eugene suggests almost nobody uses multi-million token windows, the host immediately counters from personal technical experience, asserting he uses it frequently.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Evolution of State Space Models and Subquadratic Primitives | 0 | 0 | 0 | 0 | This is an uninterrupted slide presentation by Dan Fu tracing the history of state space models, linear attention, and subquadratic primitives. Because it is a solo technical lecture without host participation, all host interaction and combativeness scores are zero. | |
| RWKV Architecture Mechanics and Open Source Origins | 0 | 0 | 0 | 0 | Eugene Cheah delivers a presentation segment outlining RWKV's open-source origin, GPU cascading mechanics, and separation into time-mix and channel-mix. There is no host involvement or adversarial tension. | |
| Transformer Conversion and Hybrid Architecture Synergy | 0 | 0 | 0 | 0 | Eugene explains model weight conversion methods (QRWKV) and the counterintuitive performance boost seen in hybrid transformer-SSM models. The segment remains a pure monologue presentation. | |
| Hardware-Model Co-Design and Next-Generation Modalities | 0 | 0 | 0 | 0 | Dan Fu discusses CUDA co-design library ThunderKittens, matrix-level primitives, and real-time streaming video generation. No host interaction occurs during this segment. | |
| Perspectives on Retrieval-Augmented Generation and Fixed-State Memory | 0 | 0 | 1 | 0 | The guests discuss their hot takes on RAG, comparing fixed-state recurrent memory to human biological constraints and external database lookups. The dynamic is cooperative between co-speakers. | |
| Audience Q&A on Context Scaling, VRAM, and Streaming | 4 | 3 | 2 | 3 | The host opens Q&A with a provocative pushback asking who actually needs 2-million-token contexts when RAG exists, and notes his own frequent usage of Gemini's large window. The guests acknowledge the challenge and provide nuanced answers on VRAM scaling, streaming stability, and long-range benchmarks. |
Statements from this episode (15)
Fu: Efficient AI Architectures Are Dead on Arrival Without Hardware Co-Design
“Even if your model is theoretically more efficient, if somebody goes and runs it and it's two times slower one of the things that, that we've learned is that if you're in that situation, it's just going to be dead on arrival. So you want to be designing your a…”
Fu: AI21's Jamba Is the State of the Art Non-Transformer Model
“AI-II trained this hybrid MOE called Jamba that, that, that seems, that is currently the state of the art for these non-transformer architectures.”
Fu: Stanford and Arc Institute's DNA SSM Made the Cover of Science
“One of those gated, SSM gated states-based models ended up on the cover of science because a great group of folks went and trained some DNA models. So that's Michael Polley, Eric Yuen from Stanford and the Arc Institute.”
Cheah: RWKV Trains Models on Over 100 Languages Targeting 200
“We actually train our models primarily on over a hundred language, which is another topic altogether and our goal is to train to even 200 languages to cover all languages in the world.”
Cheah: RWKV Innovations Are Found Empirically Before Academic Rationalization
“Officially in the paper, I'll say we had this idea and we wrote it this way. The reality is someone came in the code, we tested it worked, and then we rationalized it.”
Cheah: RWKV organization has less compute than a single Google researcher
“So our entire organization has less compute than a single researcher in Google.”
Cheah: Most enterprise AI workloads use 70B models under 32k context
“Majority of enterprise workload today is just on Senti B at under 32 K context line.”
Cheah: Hybrid SSM-transformer models outperform pure baselines of both
“None of us understand why a hybrid with a state-based model, the RWA state space, and transformer performs better than the baseline of both. It's like when you train one, you expect, and then you replace, you expect the same results. That's our pitch. That's o…”
Fu: Changing one PyTorch line requires a week of CUDA development
“If we decided to change one thing in PyTorch, like one line of PyTorch code is like a week of CUDA code at least.”
Fu: Modern GPU compute primitives should be matrices, not floats
“We basically built a whole library just around this basic idea that all your basic compute primitives should not be a float, but it should be a matrix and everything should just be matrix compute.”
Fu: Real-time long-context video generation cannot use quadratic attention
“You're certainly not going to do a giant quadratic attention computation to try to run that.”
Fu: Embedding model quality barely matters for final RAG performance
“We had this experience over and over again where you could have any, an embedding model of any quality, so you could have a really, really bad embedding model, or you could have a really, really good one by, and by any measure of good, and for the final RAG ap…”
Cheah: RWKV operates with a 40-megabyte fixed state size
“Like, we, like, RWKV is running at 40 megabytes for its state.”
Dan Fu: Nobody is actually submitting 2M token prompts into LLMs
“Nobody is actually putting in a two million context prompt into these models.”
Cheah: Non-positional attention architectures remain stable beyond trained context
“One key advantage of this alternate attention mechanic that is not based on token position is that the model don't suddenly become crazy when you go past the eight K training context or a million context. It is actually still stable. It's still, it's able to r…”