Eliezer Yudkowsky: commitment

Transcript context

…I think a better analogy is just put him in a high position in the Manhattan Project and say we will take your opinions very seriously and in fact, we even give you a lot of authority over this project. And you do have these aims of solving poverty and doing world peace or whatever. But the broader constraints we place on you are — build us an atom bomb and you could use your intelligence to pursue an entirely different aim of having the Manhattan Project secretly work on some other problem. But he just did the thing we told him. He did not actually have those options. You are not pointing out to me a lack of preference on Oppenheimer’s part. You are pointing out to me a lack of his options. The hinge of this argument is the capabilities constraint. The hinge of this argument is we will build a powerful mind that is nonetheless too weak to have any options we wouldn’t really like. I thought that is one of the implications of having something that is at the human level intelligence that we’re hoping to use.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.