RWKV, every mention
32 scenes, the whole family · ← back to RWKV
tap a year for its mentions
every year anyone Eugene Cheah 56Shawn Wang 4Tri Dao 2Dan Fu 1Alessio Fanelli 1
Verbatim, from the transcripts: the passages where RWKV comes up
The AI Agenda: GPT5 leaks and the business of AI News — Steph Palazzolo, The Information
- ▶ 1:10:54 Shawn Wang There are some notable exceptions, primarily together with the Mombard architecture, um, recursal with RWKV.
2024 Year in Review: The Big Scaling Debate, the Four Wars of AI, Top Themes and the Rise of Agents
- ▶ 21:06 Alessio Fanelli I think speculative decoding, we had, you know, Eugene from RWKV on the podcast before, like he's doing a lot of that with Federalist AI.
- ▶ 43:47 Shawn Wang And I think it's like quite useful, like windows as well, rolling out RWKB.
- ▶ 1:14:46 Shawn Wang It seems like things are scaling out in state-based models and RWKBs. 2 times in the scene
2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
- ▶ 0:21 Eugene Cheah I lead the art of theory team and I I'm CEO and co-founder of Federalist and we both work on this new post transformer architecture space.
- ▶ 18:11 Dan Fu And with that, I'll pass it on to Eugene, who can tell you a little bit about the latest and greatest with RWKV.
- ▶ 18:23 Eugene Cheah Uh, yeah, so I think one common question that we tend to get, uh, asked, right, is what's the difference between RWKV and state space? 8 times in the scene
- ▶ 24:50 Eugene Cheah And, and I believe together AI worked on the law cats for, for the Mamba side of things, and, and we took some ideas from there as well, and we essentially did that for RWKV. 6 times in the scene
- ▶ 33:29 Eugene Cheah Yeah, so, um, I think a lot of folks are, like, extremely excited of the idea of be it RWKB or state space potentially having infinite context. 3 times in the scene
- ▶ 38:16 Eugene Cheah Alright, so, for some people, there is use, uh, and I think, I think that's the, that might be, like, this is where my opinion starts to differ because, like, I think the big labs may have a bigger role in this because, like, even for… 2 times in the scene
[Paper Club] 🍓 On Reasoning: Q-STaR and Friends!
- ▶ 24:15 unnamed speaker Uh, do you, uh, what is the tokenizer for RWKV?
The Four Wars of the AI Stack - Dec 2023 Recap
- ▶ 35:25 unnamed speaker And then, obviously, uh, RWKV and Mamba and Stripe, Stripe Tyena from, um, together. 2 times in the scene
RWKV: Reinventing RNNs for the Transformer Era
- ▶ 0:12 unnamed speaker Uh, Eugene has been a very valued part of our Latent Space Discord for a while, and also diving deep onto RWKV.
- ▶ 18:24 Eugene Cheah So, so, so that's where I ended up landing on the RWKV model, because in that sense, right, so, so you, one thing that I always found very weird for Transformers, but, I mean, it's by design,
- ▶ 20:01 unnamed speaker Um, so we're going to go into our WQV paper in a bit, but one thing I wanted to ask, um, you kind of glossed over how you found it in the first place. 8 times in the scene
- ▶ 27:17 Eugene Cheah And I think more recently, and this model is not a hundred percent trained yet, more recently, RWKB has released what we call the world model, where we go the next step,
- ▶ 31:57 unnamed speaker What is RWKV? 8 times in the scene
- ▶ 37:52 Eugene Cheah Ok, so, uh, ok, before we go into the nitty gritties of how the layering and everything works, on the high level, right, currently, RWKB, architecturally, as a model, it can be, what we have already proven is that it can be scaled and… 3 times in the scene
- ▶ 38:24 Eugene Cheah Uh, we also have, uh, there'll be the, there'll be the, uh, RWKV Raven models. 6 times in the scene
- ▶ 38:52 unnamed speaker Okay, so there's world, there's Raven, there's music. 5 times in the scene
- ▶ 42:30 Eugene Cheah So we have the RRMTV architecture code, so that's the main model building block, and basically we feed it the data.
- ▶ 43:26 Eugene Cheah And so, so, so, so, take, for example, right, like, within, within, even for Raven and the World Model, as we go through the training stages, right, uh, we, we specifically ask people in other nationalities within our Discord community to… 5 times in the scene
- ▶ 50:39 Eugene Cheah So, um, so RRKV, uh, fundamentally it's still an input output model and you could do it for anything that you want. 4 times in the scene
- ▶ 56:21 Eugene Cheah So, so the part where we say that RWKV is a neural network that can be scaled, can be rolled out as a transformer, right? 3 times in the scene
- ▶ 1:04:12 Eugene Cheah Of, let's say, not being able to paralyze well in training, but as I, ah, as I covered, RWKB kind of solved that by splitting, effectively splitting the layers, allowing you to train different parts in parallel, ah, and, like, some people… 3 times in the scene
- ▶ 1:07:26 unnamed speaker Why isn't Artiquity a bigger deal 10 times in the scene
- ▶ 1:16:15 unnamed speaker In a way, RWKV is kind of 4 times in the scene
- ▶ 1:17:40 Eugene Cheah The world model, and subsequently some experimental models for RWKB, which is the next generation. 2 times in the scene
- ▶ 1:25:16 unnamed speaker You had one more comment about alternative models, and you, you mentioned that you actually, apart from other BKV, which is one thing, it's not like your whole identity, you're very, you're very involved right now. 6 times in the scene
- ▶ 1:34:52 Eugene Cheah They don't care if it's RWKB or Transformer underneath the hood. 5 times in the scene
- ▶ 1:45:58 Eugene Cheah Uh, it's like, it's like, um, I, I, I think those active in the RWKB, uh, discord, we know, we know a few members that literally just 2 times in the scene