RLHF

“My bet is this though, that people keep trying to get models to do things that align with their goals and alignment is an unsolved problem. We keep trying to like RLHF, like try to align these models to do the right thing. In harness space, we actually have an…”

“What we know though, is we moved from this pure, like human data, you know, that was first just pre-training on human data and then also aligning with like human preferences that was called RLHF, where we had a lot of human in the loop and human data. To like …”

“The components that you just mentioned, I'm pretty sure will be part of the final architecture for AGI. So I think they've come such a long way now and we've proven out so many things about what they can do. I can't see a world in which we will sort of realize…”

“In fact, in the Google I.O. In, in 22, We launched something called AI Test Kitchen, and that was Lambda, but we had constrained it because internally we didn't have an end-to-end version which was RLHFed, right? So the version I saw was a lot more you know, t…”

“We learned a lot about how bad our LHF is with people, just like people pay really attention to the tone, to the details, to like how many specific facts or figures on the hypothesis, right? Like actionability about like if the experiment is feasible, but what…”

“And so the only way to actually do the fine-tuning process consistently And to get it accurate for any specific context is RLHF.”

“Really, at the end of the day, like, RLHF, RLVR, They're both policy gradient methods, but the, what's different is just like the input data.”

“But if you only do SFT and the SFT data is annotated by human, then your performance is funded by human. You cannot get, kind of, superhuman performance just by, kind of, this kind of data engine approach to use human annotated data and then learn from that. Y…”

“So it was a bit of a brittle technique, but it was a bit of RL that was extremely crucial to making the models chat.”

“These reward models tend to have a lot of problems and you can over optimize them much more easily because the reward models will pick up on features that are maybe emojis or something like this that you don't actually care about where RLVR is much better matc…”

“The other problem I see is the constant reward mechanism of keep the conversation going of it always asks you a follow-up, which is not great for workflows. I want my workflow to stop, not to continuously work.”

“They can't reason about it because of all the RLHF and all sorts of limitations.”

“As humans we tend to, it's very hard to give, like, concrete score. But it's easier to do comparisons, right?”

“Now, for these verifiable domains like coding and math, instead of doing reinforcement learning with human feedback, you can do reinforcement learning. Because you can automatically check when you got the correct answer or not in these verifiable domains. And …”

“Surge is the largest player in RLHF”

“In the same way that chatbot arena can never be saturated. RLHF can never be solved.”

“Things definitely hit an excellent point with ChatGPT because I think people just saw how Incredibly valuable human data and RHF was. So definitely chat CPT was an inflection point for us, but even before that we were, we had very strong growth.”

“We found that it makes it a lot easier for LLMs to follow, because a lot of elements were post-trained in RLHF with kind of XML type of input, and it turns out to produce better results.”

“One of the things that's known a lot is Claude is sort of the more happy and more human steerable model, and the other one is Lama. Four is one that needs a lot more steering. It's almost like talking to a developer, and part of it could be an artifact of not …”

“GRPO is, like, great for, like, leaning heavy on highly parallel inference compute. It's more memory efficient for the actual training process. It's much easier to do in a distributed fashion because you have less gradient syncing and less model weight copies.”

“Scale partnered with open AI very, very early on before chat GPT came out. This was like very early on RLHF when they were trying to tune models to summarize better based off of Reddit passages. And this is on GPT two.”

“First step is you go to an RLHF, RLXF, whatever framework you have. You turn off the reward term. So you delete the reward part of it. All you are left is some KL term. And now you swap your KL, the anchor policy, to a bigger teacher policy. And there you go. …”

“You can actually combine RLHF and distillation together, because now you're doing two things at the same time.”

“I think that caught pretty much everybody, but the people in OpenAI by surprise was that you can have a relatively small number of examples from people, fine tune the model on that, and then it's a lot easier to work with.”

“You have just the basics, but then when it comes to, like, ChatGPT or GPT Instruct or Cloud, No one published the details there. And so we had to reinvent the wheel there in a very short amount of time.”

“Having bigger models enables to collect better data, for instance, at RLHF stage, because that's the model we use for the annotation. And so we distillate straight forward, like those annotations from this better model to the other models. So I can guarantee y…”

“And because of that, you can have a model that flats the bad outputs, and learns to only shift towards the best and better and better outputs. And you can even end to superhuman abilities, since that I'm bad at writing a poem, but I'm good at judging which one…”

“It was the RLHF and the dialogue component of ChatGBT that triggered the takeoff. And that was the capability threshold. Like it was at that time that this capability threshold passed where now it is a prior, like LLMs are a priority for every organization.”

“I think it's, I try to do it every six or 12 months is my current, is my estimated cadence, just to refine the ways that I say things, and people will see That we don't know that much more, but we have a bit of better way of saying what we don't know.”

“I think any major company that wasn't doing RLHF is now realizing they have to have a team around this.”

“At the same time, we don't have a lot of that in the like open and research communities at the same scale.”

“Everyone knows Anthropik is kind of the masters of this”

“In RLHF calling things a preference model is a little annoying because There's no inductive bias of what a preference is.”

“The RLHF techniques that people use were built in, like, labs like OpenAI and DeepMind, where there are some of these people, they have, they, these places do a pretty good job of trying to get these people in the door when you compare them to, like, startups,…”

“Setting up the infrastructure to take tens of thousands of prompts and generate them and then show them to a human and collect the human responses and then show that Shove that into your training architecture is very far away from working, so we don't really h…”

“What happens is the state is a prompt, and then you do a completion, and then you throw it away, and you grab a new prompt. Where really in like RL, you, as an RL researcher, you would think of this as being like, you take a state, you complete, Get some compl…”

“As reinforcement learning is so much less compute, like it, Is a richer signal in terms of its impact, because if they could do what RLHF is doing at pre-training, they would, but they don't know how to have that effect in like a stable manner. Otherwise, ever…”

“I think for most people, instruction tuning is probably still more important in their day-to-day life. I think instruction tuning works very well. You can write samples by hand that make sense. You can get the model to learn from them. You could do this with v…”

“I don't really recommend most startups to do it unless it's like going to provide them a clear competitive advantage in their kind of niche. Yeah. Because you're not going to make your model ChatGPT like better than OpenAI or anything like that.”

“The KL constraint prevents that. There's not that much documented work on that, but there's a lot of people that know if you take that away, it just doesn't work at all.”

“The reason this really is done on a deep level is that you're not actually trying to model any, like, contestable preference in this. Like, you're not trying to go into things that are controversial or anything. It's really, like, the notion of preference is t…”

“I don't really think this disagreement data is used for anything, but it's good to know, like, what the distribution of prompts is, who's doing it, how many samples you have, controlling the workforce.”

“You could use like a hundred times smaller language model and do much better at filtering than RLHF”

“I think Zephyr was the first model that showed success with RLHF in the public, but that's a long time from everyone knowing that it was something that people are interested in to having any, like, check mark.”

“It really ends up being kind of, like, gibberish that I think is less important now, because it's more about data and infrastructure than RL details, than, like, value functions and everything. A lot of the papers have different terms in the equations. I think…”

“It's like, now you're getting to the point where you don't even really need this to get a good model, so that's why it's like, okay, the RL is such a small part of the actual, like, doing RLHF. Like, RLHF is a metaphor for, like, all language model adaptation,…”

“RLHF is not that shown to improve capabilities yet. I think one of the fun ones is from the GPT-IV technical report. They essentially listed their kind of bogus evaluations, because it's a hilarious table, because it's like LSAT AP exams, and then like AMC-X a…”

“PPO is kind of a nightmare to scale.”

“There's a few papers that people have published Not a lot of traction. I think it could take off. Some people that I know in the RLHF area really think a lot of people are doing this in industry just because it makes the kind of training process simpler and th…”

“Most RLHF is only a few epochs, at least in the open models”

“I think DPO is closer to RLHF than RLHF is to RL.”

“I think big labs are so over-indexed, are indexed on their own base models, so they don't know, like, what's swapping between CloudBase or GPT-IV-Base, how that would change any notion of preference or what you do with RLHF.”

“And this is like a sort of outer loop optimization that no one in the open is even remotely qualified to talk about, but OpenAI does monitor and they'll like rerun RLHF and train a new reward model with a mixture of their curated data and user prompts to try t…”

“And it's that RL loop that is very interesting because since it's programmatic, since we have an Oracle of truth, we can scale this up far larger, right? Magnitudes larger than what you can do with human feedback today.”

“Which is that why can't it be another LLM or a pipeline of LLMs that can help with that feedback? I think manual labeling is very tedious, especially for our target user, which is a software engineer. And I don't think people should necessarily have to do all …”

“I was one of the, like, co-inventors of that at OpenAI, but since then it's been, you know, improved to power ChatGPT”

“One thing that I've seen with significant research labs is like still continued insourcing of annotators for both pre-training sets and LHF because some of the external services and marketplaces can't get to the level of quality that they're looking for in par…”

“GPT-IV is not good because it's large. It's good because it uses reinforcement learning from human feedback, which was discovered in the StrapGPT paper two or three years ago.”

“I don't like the RLHF models. I don't like the tuned versions of them. I think that they become, you take on the personality of a customer support agent, right?”

“Full blown, like RLHF is, you know, definitely something we're going to look into that, but there is like several many steps that you can have in between that significantly can increase your quality. So for example, I mean like even using something like a reje…”