Dwarkesh Patel: commitment

Transcript context

…So what you’re saying is that if you go to Oppenheimer and you say, “Here’s the genie that actually does what you meant. We now gift to you rulership and dominion of Earth, the solar system, and the galaxies beyond.” Oppenheimer would have been like, “Eh, I’m not ambitious. I shall make no wishes here. Let poverty continue. Let death and disease continue. I am not ambitious. I do not want the universe to be other than it is. Even if you give me a genie”, let Oppenheimer say that and then I will call him a corrigible system. I think a better analogy is just put him in a high position in the Manhattan Project and say we will take your opinions very seriously and in fact, we even give you a lot of authority over this project. And you do have these aims of solving poverty and doing world peace or whatever. But the broader constraints we place on you are — build us an atom bomb and you could use your intelligence to pursue an entirely different aim of having the Manhattan Project secretly work on some other problem. But he just did the thing we told him. He did not actually have those options. You are not pointing out to me a lack of preference on Oppenheimer’s part. You are pointing out to me a lack of his options. The hinge of this argument is the capabilities constraint. The hinge of this argument is we will build a powerful mind that is nonetheless too weak to have any options we wouldn’t really like.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.