Aug 31, 2023 · 1h 56m · latent-space
RWKV: Reinventing RNNs for the Transformer Era
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this extensive discussion, UIlicious CTO Eugene Cheah details the architecture, grassroots open-source origins, and multilingual capabilities of RWKV, demonstrating how linear-time RNNs can match Transformer performance while drastically reducing compute and memory overhead.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
The hosts as informed peer 5.8 Guest teaching 6.1 Guest disagreement 1.5 The hosts pushing back 2.4
speaking balance: gold is the hosts, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 1:08:15 Blunt admission of marketing failure
Eugene dismisses academic rationalizations and bluntly admits that the RWKV community simply sucks at marketing compared to rival model creators.
Hardest push from the hosts ▶ 1:31:00 Host challenges text diffusion latencyThe host rejects Eugene's text diffusion concept, arguing that streaming autoregressive tokens is essential for acceptable user experience.
Biggest teaching moment ▶ 33:20 Educating the host on Attention-Free TransformersEugene introduces Apple's Attention-Free Transformer paper to explain how RWKV eliminates multi-head attention entirely, catching the host completely off guard.
The host holds their own ▶ 1:42:50 Host cites Justine Tunney's GGML mmap breakthroughThe host immediately identifies and contextualizes Justine Tunney's contribution of memory mapping to llama.cpp/GGML to prove his grasp of systems-level ML optimization.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | The hosts as informed peer | Guest teaching | Guest disagreement | The hosts pushing back | Why |
|---|---|---|---|---|---|---|
| Eugene Cheah's Background and the Creation of GPU.js | 6 | 5 | 1 | 2 | The host inquires into Eugene's background with GPU.js and AST compilation to WebGL shaders. Eugene explains the origins and mechanics of running JS linear algebra on GPUs. | |
| Building UIlicious and Applying Code Models for Automation | 6 | 4 | 1 | 2 | Eugene outlines building UIlicious and training CodeGen models for web automation. The host notes licensing nuances and the transition from pre-transformer dataset scales to foundation fine-tuning. | |
| The Context Window Bottleneck and Need for Alternatives | 6 | 6 | 2 | 3 | Eugene discusses hitting quadratic scaling walls when tokenizing full HTML pages for UI testing. The host questions tokenizer design and human working memory limits. | |
| Grassroots AI Communities and Multilingual RWKV Adoption | 6 | 5 | 1 | 2 | Eugene explains discovering RWKV on grassroots forums and why non-English communities adopted it over US-centric models. The host connects this to uncensored and character AI communities. | |
| Deconstructing RWKV Architecture and Attention-Free Mechanisms | 5 | 7 | 2 | 3 | Eugene details how RWKV discards multi-head attention using Apple's Attention-Free Transformer (AFT) paper. The host expresses surprise at attention-free transformer implementations. | |
| RWKV Model Lineage and Multilingual World Datasets | 5 | 6 | 1 | 2 | Eugene walks through the model lineup including Raven, World, and Pile Plus. The host asks about data provenance, GPT-4 dataset scrubbing, and multilinguality ratios. | |
| Human Evals and Redesigning Tokenization with Tries | 5 | 7 | 2 | 2 | Eugene breaks down RWKV's custom trie-based tokenizer that eliminates space-delimited assumptions for Asian languages. The host probes dictionary size and character-level capabilities. | |
| Multimodal Capabilities and the Genesis of Blink's Research | 4 | 7 | 2 | 2 | Eugene outlines multimodal RWKV experiments in vision and MIDI music alongside Bo Peng's (Blink) history. The host explores how an independent researcher secured compute backing from Eleuther and Stability. | |
| Parallel Training Mechanics: Time-Mix and Channel-Mix Explained | 6 | 7 | 1 | 2 | Eugene illustrates how time-mix and channel-mix allow layer cascading to saturate GPUs without sequential bottlenecks. The host notes parallels to recurrent networks and Big-O complexity. | |
| Linear Computational Complexity and Low-Resource Edge Deployments | 6 | 6 | 1 | 2 | Eugene explains O(1) inference memory savings and edge execution on Raspberry Pis. The host probes parameter memory vs working context memory footprints. | |
| Adoption Roadblocks, Context Scaling Limits, and RWKV v5 | 7 | 6 | 2 | 4 | The host presses Eugene on why RWKV hasn't gained mainstream mindshare. Eugene points to marketing deficiencies and context degradation on long document QA, outlining RWKV v5 solutions. | |
| Decentralized Open-Source Governance and the RWKV Foundation | 6 | 5 | 1 | 2 | Eugene describes the self-organizing Discord channels and plans for an RWKV non-profit foundation. The host offers to connect the team with non-profit organizational resources. | |
| Text Diffusion Models and Tackling the Token Crisis | 6 | 7 | 2 | 4 | Eugene pitches applying diffusion to text models to enable multi-epoch training and bypass the token crisis. The host pushes back by citing streaming generation latency and UX requirements. | |
| Engineering Pathways in AI: From Prompting to Systems | 7 | 6 | 2 | 2 | Eugene maps out learning paths from prompt engineering to low-level systems programming. The host and Eugene discuss software engineers making high-impact ML contributions via mmap and CUDA memory alignment. | |
| Character Alignment and Optimization in Persona-Driven AI | 6 | 7 | 2 | 2 | Eugene discusses character persona alignment and why LoRAs excel at persona speech patterns rather than net-new knowledge. The host draws connections to personality modeling and digital consciousness. |
Statements from this episode (30)
GPU.js Outperforms V8 on Matrices Over 2,000 Dimensions
“It outperformed the base VA engine by running it on the WebGL. Well, especially when you scale past 2000 dimensions there is a gotcha because you have to transfer your variables from the JavaScript space to the GPU space. So anything less than a thousand by th…”
Fine-Tuning Transformers Requires Only 1,000 to 10,000 Data Examples
“Post-Transformer it's literally, you probably need only, like, a thousand or 10,000, enough, like, data that you can literally get an intern a few weeks to just get it done, and you have a working model. It may not be that great, but frankly, every piece of da…”
Cheah: Standard Transformers Will Never Scale to Ten Million Tokens
“I think what was quick, I think it was rather quick after I concluded that transformer as it is will not scale to ten million tokens.”
Most Open-Source LLMs Fall Flat Outside English-Speaking Nations
“Beyond OpenAI's model, and beyond ChatGPT and Claudia, the two big models, right? Outside of the English speaking nations, right? A lot of the open source models really fall flat.”
Foreign Language Data Degrades English Benchmark Scores on Small LLMs
“Adding in a foreign data set is actually a loss, because once you're below a certain param count, so we're talking about the seven important, right? The more you add that's more in line with your evals, the more it will degrade, and they just exclude it.”
RWKV Matches GPT-NeoX Performance at Equal Parameter and Data Scales
“RWKV is a modern recursive neural network with transformer-like level of LMM performance, which can be trained in a transformer mode. And this part has already been benchmarked against GPT-NeoX in the paper, And it has similar training performance compared to …”
RWKV Architecture Is Proven to Scale to Any Parameter Size
“What we have already proven is that it can be scaled and trained by a transformer.
How I do so, we'll cover later.
And this can be scaled to as many parameters as we want.”
RWKV Raven Dataset Scrubs Out 'As an AI' Refusal Boilerplate
“Typically GPT for all, but then we scrub it for and remove all the, as a large model.”
RWKV Prioritizes User Feedback Over Benchmark Evals for Dataset Additions
“The reason why we add things to the data set was never about improving evals. It's about directly in response to user feedback.”
RWKV Relies Exclusively on Native Speakers for Multilingual Quality Evals
“Our informal rule is that, the only person who can decide if whether this Improved word model is better in Japanese, or Thai, or whatever it is, it's a native speaker.”
GPT-NeoX Documentation Became Reference Notes for Subsequent Open-Source LLMs
“GPT Neo X was that it was one of the major models that had everything fully documented and they like, why they make this change in the architecture and so on and so forth. And that became like Basically reference notes for all other subsequent open source mode…”
RWKV Uses Trie Tokenizer Without Space Delimiters for CJK Languages
“Instead of using like this token pairs well with this and should be paired with that we just made it a trial list. So So basically, try the data structure. Yeah. So we just find the longest matching string in that matching string that we have trained inside ou…”
RWKV Community Built Multimodal Vision Models Using MiniGPT-4 Architecture
“There is actually another project internally on the discord where it's Doing vision, ah, vision modeling, and this based on the, ah, is it, mini GPT-Fall paper, where, where you have an image model, put everything inside the latent space, and then you have the…”
Bo Peng Created RWKV on EleutherAI Forum to Parallelize RNNs
“Blink, or Bopeng is the actual name decided basically as an individual, literally at the illiterate AI forum, decided that, hey I think we can modify recurrent neural networks, no, neural networks, based on the Apple paper, the light attention that I showed pr…”
EleutherAI and Stability AI Donated A100 Compute to Train RWKV
“So, so that's why AI and the rest stability, I believe also is involved, stepped up. And donated the A-One-Hundreds needed to train the basic models that RWKB had.”
RWKV Trains in Parallel Across GPUs, Unlike Traditional RNNs
“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”
RWKV Inference Computes at O(1) Complexity Per Token
“I'm talking about, like, to go through the entire context, yeah, this will be O one per token.”
RWKV Token Inference Only Requires Current and Next States in RAM
“If you really, really want to, like, save RAM, You, it is possible for you to do token by token inference, so that you don't need to keep your states in history. You only need to keep your current token state and your next.”
Cheah: RWKV Achieves Linear Scaling With No Trade-Offs in Reasoning
“So, so this is like literally us saying, there's no trade-offs. Yeah, you don't lose out in that process.”
Microsoft and Other Research Labs Cite RWKV in Architecture Papers
“Ever since that initial paper came out, there was ResNet, there's I think there's two more, there's a few more additional papers coming out, one from Microsoft, one from other organizations that are literally exploring the whole idea, once again, of scalable n…”
Half of the RWKV Community Joined for Non-English Multilingual Support
“The only reason why we have a bigger outsized impact compared to like the other models is frankly because half of our discord
Came in not for English.
It's for other languages.”
RWKV Performance Degrades on Context Lengths Beyond Its Training Data
“Well, it will, as a neural network, it will happily keep going on for infinite context, man.
It will just keep generating.
does it do well?
That's the answer is no, because if you didn't train it to handle that situation, and that's actually a child rule.
So,…”
RWKV Creator Intends to Build the AI Equivalent of Linux Foundation
“He seems to be heavily inspired and wants to go towards the direction of creating the equivalent of a Linux foundation for AI models. So he really wants this to be open source.”
Training LLMs Beyond Two Epochs Causes Overfitting and Degradation
“Anything beyond that, and we can confirm, even for our model, ours is more like closer to two, but the idea is still there, that it starts to overfit, and it starts to degrade in a lot of things.”
The Token Shortage Crisis Only Applies to AGI, Not Small Models
“I would say if we are aiming for AGI, there is a token crisis, but if we are aiming for useful small models, I don't think there is a token crisis.”
Cheah: AI Engineers Do Not Need ML Math to Build Products
“Frankly, for an AI engineer, you don't need it. You, your main thing that you needed to do was to, frankly, just play around with ChatGPT, or all the alternatives, be aware of the alternatives, because be very mercenary, swap out to Cloudia if it's better for …”
Cheah: Pre-Transformer Academic Neural Network Research Is No Longer Relevant
“Frankly, almost everything that is, that matters, Ah, was basically in the past four years. Like, there were a lot of things that fit in academics that were before that, and you know, and they were mostly dealing with models that were under a billion parameter…”
ML Researchers Optimize for Working Models Over Code Efficiency
“A lot of ML scientists, when they really build this stuff, the focus was more of like, always get it to work. It was never about getting it to work efficiently, or getting the code documented or organized.”