The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 12 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
RWKV Architecture Is Proven to Scale to Any Parameter Size
“What we have already proven is that it can be scaled and trained by a transformer.
How I do so, we'll cover later.
And this can be scaled to as many parameters as we want.”
RWKV Matches GPT-NeoX Performance at Equal Parameter and Data Scales
“RWKV is a modern recursive neural network with transformer-like level of LMM performance, which can be trained in a transformer mode. And this part has already been benchmarked against GPT-NeoX in the paper, And it has similar training performance compared to …”
RWKV Trains in Parallel Across GPUs, Unlike Traditional RNNs
“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”
RWKV Uses Trie Tokenizer Without Space Delimiters for CJK Languages
“Instead of using like this token pairs well with this and should be paired with that we just made it a trial list. So So basically, try the data structure. Yeah. So we just find the longest matching string in that matching string that we have trained inside ou…”
Bo Peng Created RWKV on EleutherAI Forum to Parallelize RNNs
“Blink, or Bopeng is the actual name decided basically as an individual, literally at the illiterate AI forum, decided that, hey I think we can modify recurrent neural networks, no, neural networks, based on the Apple paper, the light attention that I showed pr…”
RWKV Inference Computes at O(1) Complexity Per Token
“I'm talking about, like, to go through the entire context, yeah, this will be O one per token.”
EleutherAI and Stability AI Donated A100 Compute to Train RWKV
“So, so that's why AI and the rest stability, I believe also is involved, stepped up. And donated the A-One-Hundreds needed to train the basic models that RWKB had.”
RWKV Token Inference Only Requires Current and Next States in RAM
“If you really, really want to, like, save RAM, You, it is possible for you to do token by token inference, so that you don't need to keep your states in history. You only need to keep your current token state and your next.”
Training LLMs Beyond Two Epochs Causes Overfitting and Degradation
“Anything beyond that, and we can confirm, even for our model, ours is more like closer to two, but the idea is still there, that it starts to overfit, and it starts to degrade in a lot of things.”
Cheah: RWKV operates with a 40-megabyte fixed state size
“Like, we, like, RWKV is running at 40 megabytes for its state.”
Microsoft and Other Research Labs Cite RWKV in Architecture Papers
“Ever since that initial paper came out, there was ResNet, there's I think there's two more, there's a few more additional papers coming out, one from Microsoft, one from other organizations that are literally exploring the whole idea, once again, of scalable n…”
RWKV Community Built Multimodal Vision Models Using MiniGPT-4 Architecture
“There is actually another project internally on the discord where it's Doing vision, ah, vision modeling, and this based on the, ah, is it, mini GPT-Fall paper, where, where you have an image model, put everything inside the latent space, and then you have the…”