Dwarkesh Patel: belief

Transcript context

…eric spruce tree is. This is not specific to the example that you gave. It’s me being like — “Well, suppose we took a criterion that sounds kind of like this and asked, how do we actually maximize it? What else satisfies it?” You’re trying to argue the AI into doing what you think is a good idea by giving the AI reasons why it should want to do the thing under some set of hypothetical motives. But anything like that if you optimize it on its own terms without narrowing down to where you want it to end up because it actually felt nice to you the way that you define niceness. It’s all going to have somewhere else, somewhere that isn’t as nice. Something maybe where we’d sooner scour the surface of the planet with nuclear fire rather than let that AI come into existence. Though I do think those are also probable because you know, instead of hurting you, there’s something more efficient for it to do that maxes out its utility function. Okay, I acknowledge that you had a better argument there, but here’s another intuition. I’m curious how you respond to that. Earlier, we talked about the idea that if you bred humans to be friendlier and smarter. Eliezer Yudkowsky 3:19:29 I think I want to register for the record that the term breeding humans would cause me to look askance at any aliens who would propose that as a policy action on their part. All right, there I said it, move on. No, no. That’s not what I’m proposing we do. I’m just saying it as a sort of thought experiment. You answered that we shouldn’t assume that AIs are going to start with human psychology. Okay, fair enough. Assume we start off with dogs, good old fashioned dogs. And we bred them to be more intelligent, but also to be friendly. Well, as soon as they are past a certain level of intelligence, I object to us coming in and breeding them. They can no longer be owned. They are now sufficiently intelligent to not be owned anymore. But let us leave aside all morals. Carry on. In the thought experiment, not in real life, you can’t leave out the morals in real life.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.