reinforcement learning

“But it's another thing if you import those same techniques into robots. If you have your robots doing reinforcement learning, which is basically trial and error learning, they start knocking over the furniture a 100,000 times, you're probably gonna send it bac…”

“Instead of trying to predict what people would like, let's, let's, let's run experiments in real time and see what people click on. And that turned out to generate very good click through rates.”

“Where this ultimately went then was kind of a system that, that Brought man and machine together, where the editors would propose anywhere from 20 to hundred, and over time, 200 different stories that they think might be good. And the reinforcement learning wo…”

“When you train large language models with reinforcement learning, they become jagged in the sense that they become good at what you wanted them to be good at. And there are some generalization capabilities, but they're much weaker than people think.”

“Give it math questions, tell it whether it got them right or wrong, and the model will learn. This is, it comes down to a bit of lesson in scale and search, is just allow the model to search, have enough compute to run the experiments, and the model actually e…”

“Test time compute lets you do harder problems than you can currently do, than you can like currently do off the cuff, and RL Then allows you to sort of distill that back into the model.”

“One way of thinking about this is test time compute is doing a lot of reasoning, and then RL is the feedback signal on whether or not that reasoning was right or wrong.”

“There's now this two paradigm regime where previously you did free training scaling and reinforcement learning scaling, and now we're in a mix of the two basically. And so I think that gives you more opportunities to update models because it means that you can…”

“Saying, I don't know, or solving, you know, hallucinations is, ah, intrinsically requires reinforcement learning in many ways.”

“One of the craziest things about RL on language models in the, in, like, the RL from verified rewards regime, is it's almost the simplest possible thing. It's, like, almost too simple to work. And this is, again, comes back to that question of taste, where Rea…”

“So that allows you to take a mid-tier model and make it as good as a larger-tier model of six months ago or three months ago.”

“In a good way, the good way to do RL is if you balance those things. So if you kind of give cookies half of the time and punish the other half of the time, but this is almost like a mathematical kind of kind of aspect of it.”

“Like whenever, even when I started like 20, 2019, the reinforcement learning was kind of fashionable at that moment, although not like very successful, but we were, the reinforcement learning was able to solve a lot of games, but the bottleneck was there that …”

“And like, I don't like in terms of a pure RL, I don't think like really pure RL makes sense. RL needs Pre-training to be successful. And I think pre-training, as I said before, needs RL to be successful as well.”

“In some way you can say it's a limitation of reinforcement learning, but when I was thinking about it, I realized a lot of that happens in human systems as well. There are a lot of like incentive system and reward systems and even, even happens in workplaces a…”

“The first thing that is important to know and understand, RL is hard. Like, conceptually, if you think about it, and there's still a lot of depth to it, but very conceptually, mathematically speaking, pre-training is dead simple.”

“Language models do on their own, like fundamental level is they are often called as next token prediction machines. And that's not completely accurate in the age of reinforcement learning, but they still operate on mostly on tokens that are mostly text.”

“And that's kind of, like, the only way how to, like, really teach agents to, like, learn to react to changes in the environment is through reinforcement learning.”

“Our pre-training data is not very agent-like. If you think of the pre-training data, right, there is like websites and books and, you know, all kinds of recent text that has a lot of information, but it doesn't have a lot of actions. It doesn't really capture …”

“If you're not careful with RL, you can make interpretability harder. For example, one Common thing with modern models is they do reasoning with the chain of thought. You could look at the chain of thoughts to, you know, see what are the model internal thoughts…”

“Figuring out what are the best reward sources, and how do we scale it up, and how do we, you know, get more rewards, more reliable rewards. That will be one of the key ingredients in scaling up RL further.”

“Yeah, I wouldn't view it alignment adjust like an RL problem. I think it sort of, it goes throughout the whole stack. You might, you know, for example, filter the pre-training data in some way. You might, after training, you might have classifiers that, you kn…”

“I think we don't have any good scaling laws yet. That tell us the trade off, especially I think because it's very hard to measure what is the quality of a data point, right? Like how good is this example compared to this other example without being able to mea…”

“Personally, I think that's unlikely. Not, not because pre-training is strictly necessary. I think we may well be able to train something completely from scratch, as we've been able to do in other domains, but more because pre-training on this vast data sets th…”

“If you look at all the RL literature over time, we see very similar returns on compute in pre-training and in RL, where we can invest exponentially more compute in RL and keep getting benefits.”

“One direction of scaling RL and making it more stable is by improving this by, for example, putting more reasoning into your language model to generate much more high quality training data. That can then give us training that is much more stable, and then we c…”

“A better base model and a bigger base model is much easier to improve with RL.”

“I do believe the era of tomorrow will be broader. It will work on general data and maybe then it will expand to like domains that, that go beyond where, where it shines today.”

“I don't work on post-training and it certainly has a lot of quirks, but I think the main part is, is indeed RL where you say, okay, is this response cynical? Is this response like that? And you say, okay, if you were told to be cynical, this is how you should …”

“And I think we are at the stage where if we define a benchmark and we can make a relevant RL environment, then we can kind of max it out pretty quickly, and so we are going through benchmarks now very, very quickly.”

“And I think that we shouldn't do pre-training. We shouldn't try to just only train from scratch. I think we're kind of focusing on post-training reinforcement learning can potentially get us better performance gain.”

“RL is now the foundation of really all post training. It's all done by RL.”

“Part of it is the model knowing when it's going down the wrong path. But this is also something that we can that the model can be trained for with reinforcement learning is like knowing, okay, like that seems like not a great path. Let me backtrack and let me …”

“When we are training more agentic systems, you only know whether you're correct at the end of your very long rollout. So you get very little information per token of whether you were correct or not. And it's hard to say it's hard to basically do attribution. I…”

“It seems that after crossing a certain scale of models that know basically everything about the world, and what we call, like, good priors about the world, It seems that reinforcement learning just started to work, and this is not only with LMS. Robotics seems…”

“Because if you just started from reinforcement learning, it would be very inefficient. Because the problem with reinforcement learning is that you have to stumble across the right answer, basically.”

“We were able to take many of the tools that we built for these, like, verifiable reward cases, and we were able to use them more generally in on, for reinforcement on, like, real use cases, and I think that's, like, really why we're feeling that right now in, …”

“When you have a lot of compute, you want to turn that compute into intelligence in a way that's useful, and RL is one way of doing it, and we just started doing it then, and we're going to do a lot more of it now.”

“I definitely think OpenAI will have amazing products that will be relevant in those domains, and some amount of RL will play a role in there.”

“If you have a powerful enough pre-trained model, then it can start to do well at RL. It can start to like think at use test time compute to for instance, solve, solve math problems that it wouldn't otherwise be able to do.”

“We do RL and we do traditional training too. Not for the largest models, for the largest lab, but for the next tier.”

“What we know though, is we moved from this pure, like human data, you know, that was first just pre-training on human data and then also aligning with like human preferences that was called RLHF, where we had a lot of human in the loop and human data. To like …”