The Ledger, every show

“I think what was quick, I think it was rather quick after I concluded that transformer as it is will not scale to ten million tokens.”

“RWKV is a modern recursive neural network with transformer-like level of LMM performance, which can be trained in a transformer mode. And this part has already been benchmarked against GPT-NeoX in the paper, And it has similar training performance compared to …”

“What we have already proven is that it can be scaled and trained by a transformer. How I do so, we'll cover later. And this can be scaled to as many parameters as we want.”

“Like this new AI architecture has the potential of reducing inference costs by over a thousand X.”

“So they are going to the direction of highly tailored sovereign AI models for the domestic market. And we actually see this happening more and more. So for Cohear, they will service the Canadian market. For the US market is going to be served by OpenAI Entropi…”

“This is the first major model that of this cell class size, right? They're saying, hey, we are doing pipeline parallelism.”

“Instead of using like this token pairs well with this and should be paired with that we just made it a trial list. So So basically, try the data structure. Yeah. So we just find the longest matching string in that matching string that we have trained inside ou…”

“And in practice, once you start cascading there, you just saturate the GPU, and that's how it starts being paralysable trained. You no longer need to train in slices like traditional RNNs.”

“You see, most providers, they only provide, let's say, less than a hundred models. That covers 50% of our inference work. It's the bottom 50% where they run all these interesting fine-tuned models that people came on board for.”

“The step one model is a particularly popular model for us that easily ship several contracts for us on this model alone. And no one else is.”

“So currently the biggest will be around one to two million dollars a year, which may sound extremely large, but when you actually peel behind the layers, it only comes to around like five, six of the largest servers you see in the market.”

“Majority of enterprise workload today is just on Senti B at under 32 K context line.”

“The crazy one, the crazy thing that we did was that we inverted the numbers during the calculation and it seems to work better.”

“One thing to warn about pricing is that you're going to see a lot of providers jumping in, and everyone's just trying to get the piece of the pie. So, so, so like with some of the previous model launches, you see some people coming in at lower and lower price,…”

“This weird scheduling, which I'm quite sure people are going to start replicating it, is to reduce the bubble, the wastage.”

“I suspect we are going to see more LoRa's first before we get full fine-tuned.”

“Beyond OpenAI's model, and beyond ChatGPT and Claudia, the two big models, right? Outside of the English speaking nations, right? A lot of the open source models really fall flat.”

“Blink, or Bopeng is the actual name decided basically as an individual, literally at the illiterate AI forum, decided that, hey I think we can modify recurrent neural networks, no, neural networks, based on the Apple paper, the light attention that I showed pr…”

“So, so that's why AI and the rest stability, I believe also is involved, stepped up. And donated the A-One-Hundreds needed to train the basic models that RWKB had.”

“I'm talking about, like, to go through the entire context, yeah, this will be O one per token.”

“If you really, really want to, like, save RAM, You, it is possible for you to do token by token inference, so that you don't need to keep your states in history. You only need to keep your current token state and your next.”

“He seems to be heavily inspired and wants to go towards the direction of creating the equivalent of a Linux foundation for AI models. So he really wants this to be open source.”

“Anything beyond that, and we can confirm, even for our model, ours is more like closer to two, but the idea is still there, that it starts to overfit, and it starts to degrade in a lot of things.”

“And Federalist was meant to be a name pricing experiment. So we gave it a different name, but within the first few days, it became more profitable and more revenue than the original company platform that we were like, I guess we are Federalist now.”

“Like, Step Fun, for example, is not an unpopular model. It's shipping billions of tokens per day.”

“Like, we, like, RWKV is running at 40 megabytes for its state.”

“It outperformed the base VA engine by running it on the WebGL. Well, especially when you scale past 2000 dimensions there is a gotcha because you have to transfer your variables from the JavaScript space to the GPU space. So anything less than a thousand by th…”

“GPT Neo X was that it was one of the major models that had everything fully documented and they like, why they make this change in the architecture and so on and so forth. And that became like Basically reference notes for all other subsequent open source mode…”

“There is actually another project internally on the discord where it's Doing vision, ah, vision modeling, and this based on the, ah, is it, mini GPT-Fall paper, where, where you have an image model, put everything inside the latent space, and then you have the…”

“Ever since that initial paper came out, there was ResNet, there's I think there's two more, there's a few more additional papers coming out, one from Microsoft, one from other organizations that are literally exploring the whole idea, once again, of scalable n…”

“The only reason why we have a bigger outsized impact compared to like the other models is frankly because half of our discord Came in not for English. It's for other languages.”