Holden Karnofsky: evaluation

Transcript context

…The one reason I might be skeptical of this theory is that I could say, “Oh, gosh, if you look throughout history, people were often convinced that they were living in the most important time,” or at least an especially important time. If you go back, everybody could be right about living in the most important time. Should you just have a very low prior that anybody is right about this kind of thing? How do you respond to that kind of logic? First of all, I don't know if it’s really true that it's that common for people to say that they're living in the most important time in history. This would be an interesting thing to look at. But just from stuff I've read about past works on political philosophy and stuff, I don't exactly see this claim all over the place. It definitely happens. It's definitely happened. I think a way of thinking about it is that there are two reasons you might think you are especially important. One is that you actually are and you’ve made reasonable observations about it. Another is that you want to be or you want to think you are so you're self-deceiving. So over the long sweep of history, a lot of people will come to this conclusion for the second reason. Most of the people who think they're the most important will be wrong. So that's all true. That certainly could apply to me and it certainly could apply to others. But I think that's just completely fine and completely true. I think we should have some skepticism when we find ourselves making these kinds of observations. At the same time, I think it would be a really bad rule or a really bad norm that every time you find yourself thinking the stakes are really high or that you're in a really important position, you just decide to ignore the thought. I think that would be very bad. If you imagine a universe where there actually are some people who live in an especially important time, and there are a bunch of other people who tell stories to themselves about whether they do, how would you want all those people to behave? To me, the worst possible rule is that all those people should just be like, “No, this is crazy, and forget about it. ” I think that’s the worst possible rule because the people who are living at the important time will then do the wrong thing. I think another bad rule would be that everyone should take themselves completely seriously and just promote their own interests ahead of everyone else's. A rule I would propose over either of them is that all these people should take their beliefs reasonably seriously and try to do the best thing according to their beliefs, but should also adhere to common sense standards of ethical conduct and not do too much “ends justify the means’ reasoning.” It's totally good and fine to do research on alignment, but people shouldn't be telling lies or breaking the law in order to further their ends. That would be my proposed rules. When we have these high stake, crazy thoughts, we should do what we can about them and not go so crazy about them that we break all the rules of society. That seems like a better rule. That's the rule I'm trying to follow. Can you talk more about that? If for some reason, we can be convinced that the expected value calculation was immense, and you had to break some law in order to increase the odds that the AI goes well, I don't know how hypothetical this would be. Is it just that you're not sure whether you would be right and so you’d just want to err on the side of caution?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.