Alexander Meinke: prediction

Transcript context

…It's the capability, also, like, you know, if you just did more pretraining, no RL. Mhmm. You know, like, we wouldn't be claiming that, like, the model becomes more reward seeking because of know what I mean? It's like, there's something specific about, like, how you train the model that I would say makes it more reward seeking. So that's so part of it is yes as it becomes more capable, but more, like, more capable with agentic RL training, for instance. Yeah. I I I think there's 2 separate claims. 1 is that if models are misaligned, then this is worse when they are more capable. Mhmm. But also that your ability to align them might get worse as they get more capable. For this paper, we are more looking at the latter. So the idea that, okay, if models are able to represent the their training process that your ability to keep them aligned to your intent as opposed to whatever their representation of the oversight system is might get worse, but that doesn't automatically imply the other claim. It's interesting as well that you're using the language of capabilities. I mean, because I I said intelligence before. So for for for me, my understanding of RL training is that it makes the models more intelligent. And for me, intelligence is the capability to acquire capability. So you put them in a novel situation, you know, the benchmarks went right up after o1. And and I felt it was because they now have this adaptation. So they've been trained to explore and to use what they know in a in a new way through some kind of composition and trial and error and and thinking and and whatnot. And and now it can adapt to novelty. So I I would say it's more intelligent, but but would you would you use the word intelligence or not?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.