Holden Karnofsky: uncertainty

Transcript context

…Gotcha. OK, let's talk about another weak point or the one you identify as a weak point. Lock-in. What do you think are the odds of lock-in given transformative AI? So lock-in is a term I use to talk about the possibility that we could end up with a very stable civilization. I talk about that in another post. It's called Weak Point in the Most Important Century. I wrote posts about the weakest points in the series and the idea is that throughout history so far, when someone becomes in charge of a government and they're very powerful and they're very bad, this is generally considered to be temporary. It’s not going to go on forever. There are a lot of reasons the world is dynamic and the world tends to just not stay that way completely. The world has changed a lot throughout history. It's kind of a dumb thing to say, but I'll get to why this might be important. If someone is running a country in a really cruel, corrupt way, at some point they're going to get old and die and someone else is going to take over. That person will probably be different from them. Furthermore, the world is changing all the time. There are new technologies, new things are possible, there's new ideas. The most powerful country today might not be the most powerful tomorrow. The people in power today might not be the ones in power tomorrow. I think this gets us used to the idea that everything is temporary, and everything changes. A point I make in the Most Important Century series is that you can imagine a level of technological development where there just aren't new things to find. There isn't a lot of new growth to have. People aren't dying because it seems like it should be medically possible for people not to age or die. So you can imagine a lot of the sources of dynamism in the world actually going away if we had enough technology. You could imagine a government that was able to actually serve everyone, which is not something you can do now, with a dictator who actually doesn't age or die, who knows everything going on, who's able to respond to everything. You could imagine that world just being completely stable. I think this is a very scary thought. It's something we have to be mindful of–– if the rate of technological progress speeds up a lot, we could quickly get to a world that doesn't have a lot more dynamism and is a lot more stable. What are the odds of this? I don't know. It's very hard to put a probability on it. But I think if you imagine that we're going to get this explosion in scientific and technological advancement, you have to take pretty seriously the idea that we could end up hitting a wall and there could not be a lot of room for more dynamism. We could have these very stable societies. What does seriously mean in terms of probability? I don't know–– a quarter, a third, a half, something like that–– I'm making up numbers. I think it's serious enough to think about as something that affects the stakes of what we're talking about. Gotcha. Are you concerned about lock-in just from the perspective of locking in a negative future, or do you think that might intrinsically be bad to lock in any kind of future? If you could press a button right now and lock in a reasonably positive future that won't have any dynamism, or one where dynamism is guaranteed but net expected positive is not, how would you make that determination?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.