The Exchanges

Q the year of RLVR and GRPO. So you had, uh, like a nice timeline where you said, uh, the year of RLVR and GRPO. So we'd love it if you could walk us through those techniques. So fair to say, so both of those belong to the world of post-training. Let's pick, uh, Uh, RLVR and, uh, let's start with the definition. What, what does RLVR mean versus regular RL?

A I would say RLHF is the biggest leap in LLMs we have seen in a long time, because that was taking GPT from GPT to chat GPT, you know, like the RLHF, the reinforcement learning with human feedback. And in that sense, uh, it's almost like LRVR, which is reinforcement learning with verifiable rewards took That other leap basically from just simple chat model to a reasoning model. Both RLHF and RLVR have the RL in it. So both are based on reinforcement learning. But I mean, this reinforcement learning is a bit different from the reinforcement learning that plays Go. It's almost like a special thing and a simpler thing in the context of LLMs. But the idea is that instead of doing next token prediction, just predicting what's the next token, it's more like looking at the full Answer. And then based on that answer, you give a reward, like, um, in RLHF it's, you have multiple answers and you say, which do you prefer? Or in the case of LRVR, you look at the full answer and then let's say it's a math problem. You say the math problem is correct. The final answer of the math problem is correct or incorrect. That's like the main difference between next token prediction and pre-training and then the RL here. So RLVR was kind of like popularized by deep seek R one, which was based on deep seek version three. And that came out R one came out January, 2025. And with that, they also introduced …

AI assessment note: “LRVR, which is reinforcement learning with verifiable rewards took That other leap”

Q the, the world of LLMs, you mentioned MOE and that triggers a question, which is what I think a lot of people are wondering, which is like, are we seeing Uh, real architecture breakthroughs within the LLM world, or are we effectively at this point polishing what we already have within the LLM world? What are you seeing that's moving the needle in terms of architecture, uh, improvement or optimization?

A Improvement is not so much coming from the architecture anymore. It is basically the post training, but like coming back to the architecture, I think it's still an interesting question because there are so many different architectures and Almost no one uses the same one. Like they're all very similar, but they are not identical. I think a lot of it is coincidental, uh, where there are some tweaks. And if you look at the loss in some cases on some training data and some, uh, training pipelines, maybe like moving the normalization, uh, the RMS norm before or after makes a small difference. I mean, there are theoretical justifications, but also for example, um, Olmo, Olmo three, which is very transparent. They moved the RMS norm, uh, placement. Um, so then Gemini had a post and pre-norm. They had both on both ends. And so there is some justification where, okay, ablation studies show this stabilizes the training, but while assuming a stable training, it's not gonna, I think, make your model magically perform better. I mean, this is just like people tune their cars a little bit by, you know, putting in different air filters and something like that. So I think it's on that level where you can make small tweaks, but it's not really changing the engine itself. The one thing though, what we've seen is a lot of Large architectures now using MOE that I think that's a new, 2025 thing. Of …

AI assessment note: “Improvement is not so much coming from the architecture anymore. It is basically the post training”

Q Fascinating. Just to unpack some of this, you can understand the reasoning steps that led to the explanation. You mentioned, um, in some of your writing label, uh, process reward models, PRMs, and the fact that this is not successful Yet. Can you unpack that, that part?

A So there's an outcome reward and a process reward. And the outcome reward is mainly like, uh, is the final answer correct or not? But then there's the whole explanation of the reasoning model, whether it leads to the correct answer. And so there's also research like, Hey, why would, why should we throw out everything the model generates and only look at the final answer? Can we get something useful out of this intermediate explanation? And the intermediate explanation is useful for several reasons. I mean, one, Is it has been shown like that this helps the model to generate the correct answer, whether the explanation is correct or not, but is a different, uh, aspect, but just the fact that it generates these intermediate steps is correlated with a more accurate answer. Then the hypothesis is if we can improve that explanation, maybe it gives even a better answer. Like even if, uh, maybe it even drives the accuracy higher. If you want to, um, like learn something, it's not enough to just see the final answer. You want to see the steps that lead to the Process reward models. They are also focused on training the model to, to reward the models based on that explanation. And so my statement that it is not so promising or useful was mainly based on the R one paper where they had a final paragraph at the bottom. I mean, this is already a year old, but they had a paragraph at the bott…

AI assessment note: “they found it wasn't worthwhile because of reward hacking.”

Q Great. What do you make of world models?

A Yeah. World models are, uh, also an interesting hot topic. So there is the whole world model aspect for more like images and physics and that stuff. So world models are basically Models that have like an internal model of the world. So they kind of simulate something internally, what you have externally. Like for example, if you have like a chess playing model, it has like an internal chess simulator built inside. So it can kind of make better predictions or predict the next states. I think that's a particularly interesting for robotics, but coming also back to LLMs. There was also a paper by Meta. It looked very promising to me as refinement or a next step for Code-based LLMs. LLMs for coding are still next token predictors. But in addition to that, what they did is they also try to predict the internal states of the variable. It's like there was like a during training and objective to, if you have a Python code to say, okay, this, at this iteration, when I, if someone would step through the code, this variable would have that and that value. And so this is in a sense, giving the model more context, more information about the training data. And it forces the model also to kind of like In quotation marks, understand training data better. So it's like, instead of just, you know, brute force, just what is the most likely next token? It has kind of like an understanding of what it…

AI assessment note: “It looked very promising to me as refinement or a next step for Code-based LLMs.”

Q you talk about tips and tricks and different techniques that triggers the thought that you had a, like a nice way of putting it in your blog post and taking a step back for a second from the weeds. But you talked about a meta lesson for all the things in 2025 and where progress actually comes from. Do you want to get into that? I think that'd be interesting.

A Yeah. And so meta lesson would be essentially here that, well, the whole, I think we are talking right now for half an hour about different things. So I think the theme would be, well, there's no one thing that fixes it all. It's a lot of little tricks and tips, uh, tips and tricks all over the place. And if you add them up, that will give you the progress. But I, yeah, I think there's no magic lever, no magic, I guess, bullet that gives you everything. It is kind of Of tweaking things here and there and making things more robust. I think the tweak was the transformer architecture back then. And now it's essentially, let's, you know, make it even better, I guess, refining it and a little bit of post training here, a little bit, maybe improving the quality and pre-training, maybe some architecture tweaks, algorithmic tweaks. It's all a little bit of everything basically.

AI assessment note: “there's no one thing that fixes it all. It's a lot of little tricks”

Q And speaking of math, I think that's one of the key questions going forward for RL, whether you can expand this beyond math encoding to other domains. What's your take on that?

A Yeah, I think what's attractive about, uh, LRVR is that you don't have to have, let's say, uh, humans checking the solutions. You have a verifier that deterministically checks For math, is the answer correct? Like giving two fractions, are the fractions the same? You know, that two numbers with decimal points, are they the same if I round them up? And so it's like very easy to check programmatically, algorithmically, uh, and the same for code. So you have code and you like code problems and you can, so in that case, they, you can compile the code. If it compiles or you have unit tests, it checks, it works. It's very nice to check. It's no, there's no subjective aspect. It's very objective. You can say, okay, it compiles. It doesn't compile. It's very, Clear cut. The question you had is, does that, uh, what happens now in general to other fields? Is it like specific to math or code? And I think we will see also expansions of that to other fields. I am personally not an expert in other fields, so I don't know what that would look like for medicine. I mean, I have like a computational biology background. I know a little bit about the drug development pipeline and so forth, but it is, I think it's not quite as clear of what the reward looks like, but You can also be more creative, so you don't, it doesn't have to be strictly verifiable through an algorithm. It can be verifiable, ma…

AI assessment note: “I think we will see also expansions of that to other fields.”

Q Are you confident in the industry's ability to keep coming up with tricks and tips going forward?

A Yeah, that is a good question. I mean, if I look at DeepSeq, for example, because, I mean, I'm, I'm always picking here DeepSeq in this podcast because, uh, I think they have a really nice trajectory of models. I wish I could also talk more about Gemini and ChatGPT, but they don't really release the, like, techniques, so hard to talk about it. So picking on DeepSeq here, I mean, if you look at a version three and then R one, and then they had version 3.2 model with the sparse attention mechanism, and then also this math version two with a self refinement and everything. So they do have right now still a track record of improving things. And they are rumored to release a new model in February, uh, the deep seek version four. But I think, well, so far, I think we are still on that trajectory where we haven't run out of ideas. So, um, I think the only thing is we are running out of is really benchmarks. So the improvement on benchmarks, it's kind of like harder to measure. And I think maybe, well, maybe it's not the one shot problem anymore where it's not really answering knowledge question. That's not really solving math problems in, in one iteration of the benchmark. It is maybe more like the agentic cycle, like where you have like a more like a objective that is not, let's say, Answer the question, but more like design something, blah, blah, blah. And then it goes off and how l…

AI assessment note: “we are still on that trajectory where we haven't run out of ideas”

Q It's fascinating, right? So back to the future, because initially people thought that they were going to train the models and then they kind of like gave up. But what you're saying is that you're seeing people going back maybe with a better state of open source LLMs that they can build on as a building block. That's what you're saying, right?

A Yes and no. I think you're right. I mean, uh, no, you bring up a good point. Uh, open weight and open source models are very, very popular, like a few years ago and are still very popular and I love working with them, but I think, well, there's still a gap between a ChatGPT model and an open source, open weight model. Maybe, you know, with DeepSeq version three, not so much, but that's almost like a different community, like the Tinkerer community, like me, like small system, uh, Well, DeepSeq version three would be way too expensive to run for me every day. I would have to spend thousands of dollars on just hosting costs every, every day, every week. And so I toy around with smaller special purpose models. But, uh, what I meant is, so first, yeah, the open source community, uh, in that sense will have maybe a comeback at these companies, but I even mean a step further that they actually develop models from scratch, like really big models. And what's different from, I would say the, The regular open source here is that it is really large scale. It's like, it's like, um, JGPD scale, uh, data center, large LM. It's not something you run on your computer, basically. It's really like a big data center style LM. So I know that there are, I mean, I can't say any names, but I know people are interested in that. Like there are, Exploring that. Whether it will work out or not, I don't k…

AI assessment note: “I even mean a step further that they actually develop models from scratch”

Q I want to come back to something you mentioned a few minutes ago, diffusion models, especially for texts. Last year, I believe Google DeepMind announced one called Gemini Diffusion. So what are those and how different are they from Transformers?

A There's of course the big field of diffusion models coming from image models, like, uh, not too long ago, maybe two or three years ago, there was like the big, uh, hype around stable diffusion, which was based on a research paper where they had like a model that Replaced, going back, generative adversarial networks, which were an idea for generating images. And so the diffusion models were essentially, instead of having a generator and discriminator set up, like two networks competing against each other, it was like a pipeline that was denoising, starting with random noise, denoising an image and coming up basically with, so with realistic looking images. And you could also Have a text prompt and basic guide in terms of it's not like a random image. You can basically guide what you want to generate. It's basically the modern generative AI image in AI that we see out there. People were wondering, okay, can we do the same thing for text? So can we use this, uh, you know, pipeline like this denoising pipeline to generate text instead of, uh, using transformers? Or, I mean, I'm saying instead of transformers, diffusion models can be transformers are often also transformers because transformers is the architecture. So LLMs Nowadays it's, um, specifically auto regressive transformers, which means these are LLMs that are generating one token at a time. So like where the next token alw…

AI assessment note: “with diffusion models, you don't have that. You have, you generate everything at once”

Q Part of where you see the world go, right? This combination of like open source model and private data. I think you call that, uh, the edge in your, in your blog post.

A What distinguishes different LLMs right now? They are all kind of similarly good. I would say like, uh, open-weight LLMs. I love LLMs that are similarly good. I mean, personally, I don't use all of them all the time, but so I usually use one LLM at a time, but like if you use or compare ChatGPT, Gemini, Um, Claude, Grock. I think they are all pretty much on the same level. Like, and I think that's because they're trying to do everything. Like the generalist models for a general person to do a lot of things. I mean, Claude is a bit more specialized to code now, but, um, the other ones, they are more like general models. And I, I wouldn't say one is significantly better than the other. They have like small, I mean, differences and so forth. But so if you want to really distinguish them and make them better in, in certain Industries. I do think, yeah, the private data is what helps, like all the treasure troves of data that a finance company has over the years, over a hundred years or 50 years collected, or medical data, like medical records from patients. I think JGPD had like a contract now to process them, to make it secure and private. But I honestly think these companies, they don't want to just give away that data first. They can't. They, I mean, it makes also sense. You just really, as a patient or a customer, I would feel really bad if someone Gives my health data to some …

AI assessment note: “if you want to really distinguish them... the private data is what helps”

Q sequel, uh, currently how to build reasoning models from scratch. Is that the, is that the, the title? And you produce, uh, an incredible amount, uh, of, of, of work. So people should find you. On, uh, your website, on your sub stack letter. How do you absorb all of this knowledge? Uh, to which extent are LMS part of your workflow? I'm just curious how you work these days.

A Good question. So I must say like, uh, well, I don't have like a magic ball approach or anything. I think I, I think the thing I have maybe is I get very excited about things and Then when I'm excited about something, it goes very easy and very fast. I don't know. It's like, well, if you notice, maybe I write only about certain topics. I don't cover image models at the moment, for example, because I am just very excited about LLMs. And then I don't, it's just, I can't help it. I get very excited, read all the things about it, write about it. And you know, that's, that's mainly, I almost go by intuition, basically what I, and I'm kind of lucky in that sense that like with my blog, what I, Find interesting. Other people also right now find interesting. So I think there's like a lucky coincidence that, uh, I honestly write only about things I find interesting. So I'm not kind of trying to force myself. Oh, I have to cover X, Y, Z because, well, it's something that should be covered. It's more like, oh, how does this, let's say recursive language model work. Let's just read the paper. And then I write about it, you know, like more like, um, yeah, getting excited about things.

AI assessment note: “I get very excited, read all the things about it, write about it.”

Q And what about small recursive models? What does recursive mean in this context?

A Uh, yeah. So there was also a big topic in 2025. There was the hierarchical reasoning model. And from that we had also another paper, uh, tiny reasoning models. And so they are interesting because they were getting very good performance for their very small size on the arc benchmark. So arc, uh, ARC is like a benchmark, almost like an IQ test, like a logic puzzle where they are like different symbols and you have to, you see a, Let's say an array of different symbols and you have to predict like, let's say what's the missing thing here in the bottom corner. And it's kind of like going a bit beyond text and beyond things usually on, on the internet. So in that sense, I think the motivation between behind this arc benchmark was to have something that really tests the capabilities of that model on something new that hasn't been shown during training, like a new task and how well the model can Take some examples from that benchmark and generalize to new tricky problems. Also different iterations of this arc benchmark to make it harder and harder and harder. Hierarchical reasoning model, it became like popular because it performed relatively well on that benchmark compared to very expensive models like Gemini, Chachupiti, and so forth. And it is a transformer architecture. And then there's the tiny reasoning model that is, I think, even simpler than the hierarchical reasoning model.…

AI assessment note: “you refine the answer over multiple iterations. Instead of just doing a one shot”

Q And you think people do that just out of largely economic incentives, like the companies need to raise more money and people need to have successful careers and therefore they want to look good. Is that the, is that the driver?

A I mean, I don't want to accuse anyone. I don't know for sure. Uh, I, I mean, I only know what is known on the internet. I read on, let's say Reddit a few times that Lama four was a separate model. So there might've been, I don't know, some company leader decisions that have led to that. I, I honestly don't know. Um, and maybe incentives getting good headlines and that stuff, but well, I think, uh, the open weight community is a pretty smart community. So it's like, I think it's not worth, worth risking something like that. And I think most people don't risk it. It's just implicit. It just happens. It's like, if you iterate too many times, it's a, it's a classic deep learning problem or machine learning problem. But the nice, the beautiful thing here was actually, it's not a big concern because it happened to all the models. So all of the models performed like five, 10% worse on this new data. It was pretty consistent. So if you were to rank those models, the ranking would still be the same. So in LLM terms, let's say ChatGPT and Gemini, let's say they cheat on the benchmarks and the models are 10% worse. But if the ranking is still seen, let's say Gemini is still better than GPT, then it's not a problem if both of them do that. So I think we have right now that in LLMs where I wouldn't say they are cheating. They are just using the data a lot. And from using the data a lot, wel…

AI assessment note: “I think most people don't risk it. It's just implicit. It just happens.”