Andrej Karpathy: belief

Transcript context

…You’re so good at coming up with evocative phrases. “Sucking supervision through a straw.” It’s so good. You’re saying the problem with outcome-based reward is that you have this huge trajectory, and then at the end, you’re trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit. Given the fact that this is obvious, why hasn’t process-based supervision as an alternative been a successful way to make models more capable? What has been preventing us from using this alternative paradigm? Process-based supervision just refers to the fact that we’re not going to have a reward function only at the very end. After you’ve done 10 minutes of work, I’m not going to tell you you did well or not well. I’m going to tell you at every single step of the way how well you’re doing. The reason we don’t have that is it’s tricky how you do that properly. You have partial solutions and you don’t know how to assign credit. So when you get the right answer, it’s just an equality match to the answer. It’s very simple to implement. If you’re doing process supervision, how do you assign in an automatable way, a partial credit assignment? It’s not obvious how you do it. Lots of labs are trying to do it with these LLM judges. You get LLMs to try to do it. You prompt an LLM, “Hey, look at a partial solution of a student. How well do you think they’re doing if the answer is this?” and they try to tune the prompt. The reason that this is tricky is quite subtle. It’s the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they’re gameable. If you’re reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed. So you can’t do this for too long. You do maybe 10 steps or 20 steps, and maybe it will work, but you can’t do 100 or 1,000. I understand it’s not obvious, but basically the model will find little cracks. It will find all these spurious things in the nooks and crannies of the giant model and find a way to cheat it. One example that’s prominently in my mind, this was probably public, if you’re using an LLM judge for a reward, you just give it a solution from a student and ask it if the student did well or not. We were training with reinforcement learning against that reward function, and it worked really well. Then, suddenly, the reward became extremely large. It was a massive jump, and it did perfect. You’re looking at it like, “Wow, this means the student is perfect in all these problems. It’s fully solved math.” But when you look at the completions that you’re getting from the model, they are complete nonsense. They start out okay, and then they change to “dhdhdhdh.” It’s just like, “Oh, okay, let’s take two plus three and we do this and this, and then dhdhdhdh.” You’re looking at it, and it’s like, this is crazy. How is it getting a reward of one or 100%? You look at the LLM judge, and it turns out that “dhdhdhdh” is an adversarial example for the model, and it assigns 100% probability to it. It’s just because this is an out-of-sample example to the LLM. It’s never seen it during training, and you’re in pure generalization land. It’s never seen it during training, and in the pure generalization land, you can find these examples that break it. You’re basically training the LLM to be a prompt injection model.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.