Insight
Cheah: Hybrid SSM-transformer models outperform pure baselines of both
“None of us understand why a hybrid with a state-based model, the RWA state space, and transformer performs better than the baseline of both. It's like when you train one, you expect, and then you replace, you expect the same results. That's our pitch. That's o…”
Insight
Cheah: Non-positional attention architectures remain stable beyond trained context
“One key advantage of this alternate attention mechanic that is not based on token position is that the model don't suddenly become crazy when you go past the eight K training context or a million context. It is actually still stable. It's still, it's able to r…”
Prediction Didn’t hold up
Cheah: Standard Transformers Will Never Scale to Ten Million Tokens
“I think what was quick, I think it was rather quick after I concluded that transformer as it is will not scale to ten million tokens.”
Insight
Foreign Language Data Degrades English Benchmark Scores on Small LLMs
“Adding in a foreign data set is actually a loss, because once you're below a certain param count, so we're talking about the seven important, right? The more you add that's more in line with your evals, the more it will degrade, and they just exclude it.”
Assertion Supported
RWKV Matches GPT-NeoX Performance at Equal Parameter and Data Scales
“RWKV is a modern recursive neural network with transformer-like level of LMM performance, which can be trained in a transformer mode. And this part has already been benchmarked against GPT-NeoX in the paper, And it has similar training performance compared to …”
Assertion Supported
RWKV Architecture Is Proven to Scale to Any Parameter Size
“What we have already proven is that it can be scaled and trained by a transformer.
How I do so, we'll cover later.
And this can be scaled to as many parameters as we want.”
Opinion
Cheah: RWKV Achieves Linear Scaling With No Trade-Offs in Reasoning
“So, so this is like literally us saying, there's no trade-offs. Yeah, you don't lose out in that process.”
Disclosure
Cheah: RWKV organization has less compute than a single Google researcher
“So our entire organization has less compute than a single researcher in Google.”
Assertion Contradicted
Cheah: Llama 3.1 405B is first frontier model using pipeline parallelism
“This is the first major model that of this cell class size, right? They're saying, hey, we are doing pipeline parallelism.”
Disclosure
RWKV Raven Dataset Scrubs Out 'As an AI' Refusal Boilerplate
“Typically GPT for all, but then we scrub it for and remove all the, as a large model.”
Disclosure
RWKV Prioritizes User Feedback Over Benchmark Evals for Dataset Additions
“The reason why we add things to the data set was never about improving evals. It's about directly in response to user feedback.”
Assertion Supported
RWKV Uses Trie Tokenizer Without Space Delimiters for CJK Languages
“Instead of using like this token pairs well with this and should be paired with that we just made it a trial list. So So basically, try the data structure. Yeah. So we just find the longest matching string in that matching string that we have trained inside ou…”
Assertion Supported
RWKV Trains in Parallel Across GPUs, Unlike Traditional RNNs
“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”
Opinion
The Token Shortage Crisis Only Applies to AGI, Not Small Models
“I would say if we are aiming for AGI, there is a token crisis, but if we are aiming for useful small models, I don't think there is a token crisis.”
Insight
Cheah: AI Engineers Do Not Need ML Math to Build Products
“Frankly, for an AI engineer, you don't need it. You, your main thing that you needed to do was to, frankly, just play around with ChatGPT, or all the alternatives, be aware of the alternatives, because be very mercenary, swap out to Cloudia if it's better for …”
Insight
Cheah: Pre-Transformer Academic Neural Network Research Is No Longer Relevant
“Frankly, almost everything that is, that matters, Ah, was basically in the past four years. Like, there were a lot of things that fit in academics that were before that, and you know, and they were mostly dealing with models that were under a billion parameter…”
Opinion
Cheah: A Human Personality and Memories Can Fit on Two SSDs
“No offense to myself, I don't think my personality and my memories is more than this. We could, even if I can exit, I could store this in two SSDs. Two hard drives.”
Insight
Cheah: RWKV Innovations Are Found Empirically Before Academic Rationalization
“Officially in the paper, I'll say we had this idea and we wrote it this way. The reality is someone came in the code, we tested it worked, and then we rationalized it.”
Assertion Not checkable as stated
Cheah: Most enterprise AI workloads use 70B models under 32k context
“Majority of enterprise workload today is just on Senti B at under 32 K context line.”
Assertion Supported
Eugene Chia: Inverting numbers in reasoning traces improves math model performance
“The crazy one, the crazy thing that we did was that we inverted the numbers during the calculation and it seems to work better.”
Prediction Didn’t hold up
Cheah: Cloud providers will slash model inference prices before raising them
“One thing to warn about pricing is that you're going to see a lot of providers jumping in, and everyone's just trying to get the piece of the pie. So, so, so like with some of the previous model launches, you see some people coming in at lower and lower price,…”
Prediction Open · timeframe Jul 2027
Cheah: AI community will replicate Meta's pipeline scheduling algorithm
“This weird scheduling, which I'm quite sure people are going to start replicating it, is to reduce the bubble, the wastage.”
Prediction Not checkable as stated
Eugene Cheah: LoRAs will arrive before full Llama 3.1 405B fine-tunes
“I suspect we are going to see more LoRa's first before we get full fine-tuned.”
Insight
Cheah: Over-quantized LLMs rapidly enter repetition loops at long contexts
“When you over-quantize, right, at longer context length, right, it starts going into repetition rapidly.”
