Reinforcement Learning

“What we have now is that the reinforcement learning type of AI technology has been put on top of the self supervised learning to get these AI models working better, which has added a level of ruthlessness to them. Because one of the things we know about RL is …”

“We are starting to see some of the labs do things like opening. I, for instance, paused some reinforcement learning for a bit on the training of its new models.”

“I think what we're going to see is a split between job functions that demand kind of mid IQ intelligence, and those will often be open weight, sort of biased with reinforcement learning, you know, things that make the models even cheaper, more performant for a…”

“I think what is definitely true is that, like, the skill of, like, developing a theory or, like, building your understanding of some poorly understood object is, like, a fuzzier one. So it might be harder, you know I guess you can try, you can tell it, you kno…”

“And then on the model side, you're looking for like world expert, reinforcement learning people who have a scientific insight on something that what you're really trying to find there is how you compare the researcher's intuition With like product thinking and…”

“If you actually have a problem that you can specialize the model around with your reasoning traces, You can start to create this compounding advantage in your domain for your customer base, where you're able to kind of shape the intelligence to be better than …”

“What we know though, is we moved from this pure, like human data, you know, that was first just pre-training on human data and then also aligning with like human preferences that was called RLHF, where we had a lot of human in the loop and human data. To like …”

“The, it's amazing how much of simulators, when you call environment.step, is still run on the CPU, and so that's usually the bottleneck for a lot of your on-policy rollouts”

“We're kind of getting to the point where you can solve basically any benchmark, right? Because what does it mean to have a benchmark? It means you've already defined the task. You've clarified what success or failure looks like. You've given a bunch of example…”

“We do RL and we do traditional training too. Not for the largest models, for the largest lab, but for the next tier.”

“And the second was that reinforcement learning was going to be the biggest driver for LLM capabilities. Today, very obvious three years ago was not an opinion held or direction held at either OpenAI or Google or Anthropic or others.”

“I have I would say a not commonly held opinion that reinforcement learning will move earlier and earlier into pre-training.”

“No doubt it's going to be better in your own harness. And it's just because of like, where are you putting your reinforcement learning compute, right? You're putting your RL and your synthetic data. You're putting it to your own harness because it's the one th…”

“And RL is batch size constraint, right? So like you are ultimately in your batch size constraint because you don't have infinite tasks, right? When you've got the entire web, you can be much more flexible in scaling up your batch size because you've got the en…”

“But what at Lilo we believe is that actually science running the scientific method and using nature and experiments as verifier is like the ultimate version of that. And so what we're building, we'll talk about these things that we call AI science factories. T…”

“And for reinforcement learning, it's always somewhere, like, around five to, like, six percent. So, said differently, that means that we're getting, like, five percent of the actual GPU computing power that we're paying for.”

“You can use things like reinforcement learning to get them really good at specific tasks that we might care about, but you do that for one task and you, it is not good at another task or it doesn't generalize.”

“RL is insanely bursty. Like when you're doing rollouts you sometimes need a 100,000 sandboxes.”

“Like the way you move around your KV cache and how efficiently you can do it, how efficiently you move your weights from your training GPUs to your inference GPUs in RL is, there's a lot of degrees of freedom, and it is basically a systems problem of Moving me…”

“Reinforcement learning is playing a huge, a much bigger role in today's kind of AI than people had maybe predicted two or three years ago that it would.”

“And if you run this reinforcement learning step on too many different tasks, the model will be able to do all of that. But once again, it's going to be very, very, very broad. And if you focus on making sure that the model understands and knows that it needs t…”

“RLs traditionally had headwinds when it's come to fields that, you know, it's more kind of, Subjective than objective. So if you kind of think of, you know, one kind of, you know example of this is creative writing, where, you know, you could take two pieces o…”

“At the very highest level, right, we have an org that focuses on pre-training, right, which is, you know, giving models a lot of world knowledge. We focus on RL, like, teaching the models how to reason with that knowledge, how to chain the little insights toge…”

“My feeling is, like customizing models is actually going to get way easier over time. That's what we're finding, because The base models are smarter, so they generate better traces in RL already, and then RL is about learning from your own past traces, and the…”

“RL, it's still taking all of this kind of Useful information from the real world, like I mentioned, all the corrections and everything, and putting it into just one number. Which is really broken.”

“Like with code or with math, it's relatively more binary, easy to verify. But how do you verify the quality of a board deck? Yeah. It's a, you have to be, you have to have like a good rubric based evaluator.”

“Coding is one of those areas where, because it's verifiable, I think that there is a good path to using reinforcement learning to improve coding models quickly.”

“In the past, it was a lot of the compute went into pre-training. Now a lot of compute goes into reinforcement learning in post-training as well. Especially after O-one came out and DeepSeek came out.”

“When you have a lot of compute, you want to turn that compute into intelligence in a way that's useful, and RL is one way of doing it, and we just started doing it then, and we're going to do a lot more of it now.”

“If you have a powerful enough pre-trained model, then it can start to do well at RL. It can start to like think at use test time compute to for instance, solve, solve math problems that it wouldn't otherwise be able to do.”

“I definitely think OpenAI will have amazing products that will be relevant in those domains, and some amount of RL will play a role in there.”

“If you want proof to be informal math, It's very annoying, because then that's, like, just makes objective function. Your code is something like Python, your proof is, say, natural language, math proof. You will not have very strong RL kind of performance, rig…”

“And this is an area that is becoming huge in terms of energy demand. It'll, it will, The probably already is bigger than what we have historically considered training, you know, pre-training”

“It will be this one 50%, yeah.”

“We were able to take many of the tools that we built for these, like, verifiable reward cases, and we were able to use them more generally in on, for reinforcement on, like, real use cases, and I think that's, like, really why we're feeling that right now in, …”

“Part of it is the model knowing when it's going down the wrong path. But this is also something that we can that the model can be trained for with reinforcement learning is like knowing, okay, like that seems like not a great path. Let me backtrack and let me …”

“Because if you just started from reinforcement learning, it would be very inefficient. Because the problem with reinforcement learning is that you have to stumble across the right answer, basically.”

“It seems that after crossing a certain scale of models that know basically everything about the world, and what we call, like, good priors about the world, It seems that reinforcement learning just started to work, and this is not only with LMS. Robotics seems…”

“When we are training more agentic systems, you only know whether you're correct at the end of your very long rollout. So you get very little information per token of whether you were correct or not. And it's hard to say it's hard to basically do attribution. I…”

“We're making a big bet on autonomy and refineries, where we use reinforcement learning to actually remove humans from the loop in determining how refineries operate.”

“If we're doing reinforcement learning on the model, it's basically inference within a sandbox with a reward function, right? And so if the model's better at more efficient inference, that RL is more efficient as well.”

“RL is now the foundation of really all post training. It's all done by RL.”

“Scale AI, Alexander Wang's company, they told me recently that most of the training that they're doing is reinforcement learning, where you build environments for the bots and they go and they try to figure out what to do.”

“So the environments can get more and more complex, and those environments run on CPUs. They don't run on GPUs. They don't run on ASICs. The ASICs run the model,”

“We came up with what I would Really say is almost the technical plan that we have pursued for the past 10 years. Number one, solve reinforcement learning. Number two, solve unsupervised learning. And number three was gradually learn more complicated, in quotes…”

“So, I think the big challenge, and this is kind of what I'm leaning up to, and what I hope to, ah, that we'll figure out here at Physical Intelligence is how to combine those threads. How to bring in all of that knowledge that you get with generative AI, but a…”

“And for example, we had this demo on, ah, making espresso. That system practiced making those espressos many, many times and used that to improve robustness, improve speed, improve throughput.”

“It turns out to be like pretty straightforward to go in and like find all those pauses and remove them. And you can speed things up further, so you can get a task where a person demonstrates what it means to succeed, and then you can have the robot practice th…”

“GRPO, for instance, it doesn't really work with any bit of policy, which was okay initially, because you are solving math problems that can be solved in like a few thousand tokens, so the model can actually generate them pretty quickly, so when you do your upd…”

“I won't be surprised if you see this happen for other forms of sort of quantitative knowledge work, just because it happens to have the properties that code has. It's testable. You know if it worked or not. It's very RL friendly.”

“I just did the story for with about scale AI saying that the majority of their training has moved to reinforcement learning where they train models to act in specific environments like filling out forms, and then they baked those capabilities back Into the mod…”

“And I think that we shouldn't do pre-training. We shouldn't try to just only train from scratch. I think we're kind of focusing on post-training reinforcement learning can potentially get us better performance gain.”

“It turns out that the mathematics that we use for diffusion models, but even for reinforcement learning, for Schrodinger bridges, for MCMC sampling, has the same mathematics as this theory, this physical theory of non-equilibrium Systems.”

“You feel like we might actually be seeing a CPU shortage partially because of this refresh cycle, but partially also because like I legitimately believe the cloud code Cloud code is increasing software creation and then on top of that, there is real demand fro…”

“Static data is becoming less important and what we might call like dynamic data that the model creates itself is, you know, for reinforcement learning is becoming more important. So, you know, I don't think data is, is, is quite the most central thing anymore,…”

“How do you get RL to work for non-verifiable domains? I think it's a pretty interesting open problem because I think that would broaden out the capabilities of the models, the improvements that you're seeing in both math and coding if we could apply those to o…”

“Long context reinforcement learning has been very, very successful in the coding world because Git has a complete history of every line of code that's been written, every comment, why it happened.”

“One is RL's kind of a shitty paradigm to learn. Karpathy obviously talks about this a lot. It takes a lot of samples to learn some very basic stuff because you only get a reward at the end. You don't actually understand things as it's happening.”

“If we want to have AI give us amazing caliber stuff. It needs to be trained on amazing caliber stuff. So, and that will change over time because we'll start, you know, we'll, we'll start that reinforcement learning around better, better quality content.”

“Pre-training or training transformers, you know on just data, like just supervised training where you just have the inputs and the outputs directly, very nice, relaxing, you know, like things are always robust, you know, things go pretty smoothly. When we do t…”