Eliezer Yudkowsky: prediction

Transcript context

…Yeah. If you start with humans, if you take humans who were raised the way Scott Aronson was, and you make them smarter, they get nicer, it affects their goals. And there’s a Less Wrong post about this, as there always is, several really, but sorting pebbles into correct heaps, describing a species of aliens who think that a heap of size seven is correct and a heap of size eleven is correct, but not eight or nine or ten, those heaps are incorrect. And they used to think that a heap size of 21 might be correct, but then somebody showed them an array of seven by three pebbles, seven columns, three rows, and then people realized that 21 pebbles was not a correct heap. And this is like a thing they intrinsically care about. These are aliens that have a utility function, as I would phrase it, with some logical uncertainty inside it. But you can see how as they get smarter, they become better and better able to understand which heaps of pebbles are correct. And the real story here is more complicated than this. But that’s the seed of the answer. Scott Aaronson is inside a reference frame for how his utility function shifts as he gets smarter. It’s more complicated than that. Human beings are made out of these are more complicated than the pebble sorters. They’re made out of all these complicated desires. And as they come to know those desires, they change. As they come to see themselves as having different options. It doesn’t just change which option they choose after the manner of something with a utility function, but the different options that they have bring different pieces of themselves in conflict. When you have to kill to stay alive you may come to a different equilibrium with your own feelings about killing than when you are wealthy enough that you no longer have to do that. And this is how humans change as they become smarter, even as they become wealthier, as they have more options, as they know themselves better, as they think for longer about things and consider more arguments, as they understand perhaps other people and give their empathy a chance to grab onto something solider because of their greater understanding of other minds. But that’s all when these things start out inside you. And the problem is that there’s other ways for minds to hold together coherently, where they execute other updates as they know more or don’t even execute updates at all because their utility function is simpler than that. Though I do suspect that is not the most likely outcome of training a large language model. So large language models will change their preferences as they get smarter. Indeed. Not just like what they do to get the same terminal outcomes, but the preferences themselves will up to a point change as they get smarter. It doesn’t keep going. At some point you know yourself especially well and you are able to rewrite yourself and at some point there, unless you specifically choose not to, I think that the system crystallizes. t keep going. At some point you know yourself especially well and you are able to rewrite yourself and at some point there, unless you specifically choose not to, I think that the system crystallizes. We might choose not to. We might value the part where we just sort of change in that way even if it’s not no longer heading in a knowable direction. Because if it’s heading in a knowable direction, you could jump to that as an endpoint. Is that why you think AIs will jump to that endpoint? Because they can anticipate where their sort of moral updates are going?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.