Nathan Lambert: evaluation

Transcript context

…if you start like expandiating both sides and taking the log of both sides, both the two papers end up being the same thing. And people don't know how to train preference models particularly well now. I think if you zoom into any of the details to look at like the agreement number, so how if you look at a test set, you'll have a chosen and rejected and you can take the reward model you're training, pass in those completions and you see if the chosen predicted reward, so the scalar number is higher than the rejected predicted reward. And this is the agreement numbers in all of these datasets is like that where you see they have the 65 to 75% agreement. This just means that like these scalar numbers were ordered correctly. And that's a pretty low number. It's not gonna get to a hundred percent. That goes to show the kind of like deep questions at play here. People are playing with different loss functions and samples, different models to try to address this, but it's really a fundamental issue. It's like, it goes back to like, what does it mean to do RLHF? And we're not gonna answer that now, but it's good to know that like this 65 to 75% agreement, you'll see these numbers everywhere. It's like, we don't have a hundred percent agreement with the reward model and the data and that's fine. That's just where we're at. And we essentially take this model and then we start throwing RL at it. I think PPO, proximal policy optimization, it's pretty complicated compared to what you really need to know. It really just does RL under the hood. Things like PPO, it learns a value function and then it uses the value function to update the model. If you actually look at like a feedback diagram, more of like a systems problem than an RL problem. So you'll see things like you need to have two copies of the language model. This is for the KL constraint that we talked about before. You need to have the reward model, which is either a separate reward model or value head on your base model. And then you need to have your like RL code that actually learns a value function and updates all the parameters. I think it just is really messy to actually set up, but if you dig into it, most people could understand what each of the components are. And then the hard parts are like, how do we actually make a language model that works out of this? Which is not something that people know that well. I think things that I talk about a lot, it's just like, okay, like what is the signal flow? How do you access the reward model? The reward model is used in RLHF exactly what you would think. You have a prompt, the language model generates a completion and then that completion is given a score. That score gets plugged into the whole RL stuff and it updates the parameters. That's kind of the core of it. There's a lot of different things zooming in on where exactly you put this distance penalty between the base model and the RL model. Most people say that you just deduct it from the reward. So if you go all the way back to RL as an agent acting in the world, the reward from that world would be a combination of the reward model and any constraints like KL that you put on it. m the reward. So if you go all the way back to RL as an agent acting in the world, the reward from that world would be a combination of the reward model and any constraints like KL that you put on it. There's a lot of different ways to do this because a lot of RL algorithms like PPO actually have a KL constraint built into them. So it's confusing because you hear KL twice, but those are different KLs. One of them is about the text and one of them is about the value function distance or the policy distance or something like this. So those are different. It really ends up being kind of like gibberish that I think is less important now because it's more about data and infrastructure than RL details, than like value functions and everything. A lot of the papers have different terms in the equations. I think InstructGPT does something where they like try to get the RL model to match the instruction tuning dataset because they were really happy with that dataset to constrain the distribution. LLAMA does some different things, but I think these are all small gains over just getting the deep understanding of the data in the infrastructure setup. This is why we say it's like so little RL. It's like now we're getting to the point where you don't even really need this to get a good model. So that's why it's like, okay, the RL is such a small part of the actual, like doing RLHF, like RLHF is a metaphor for like all language model adaptation and RL is one tool used at one point in the time. So that's kind of where I wrap up like the core overview in my mind to say like RL doesn't really do as much as people think, but you could put up flashy equations and do all sorts of stuff if you want to. It's just like, I think it's kind of misleading even because I don't think about those equations on a regular basis. But what if we call it Q star?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.