Holden Karnofsky: belief

Transcript context

…Gotcha. I and many other people are going to find that very wild. Could you walk us through the process by which you went from working in global development to thinking this way? In 2014, for example, you had an interview or a conversation and this is a quote from there. “I have looked at the situation in Africa, have an understanding of the situation in Africa, and see a path of doing a lot of good in Africa. I don't know how to look into the far future situation, don't understand the far future situation, and don't see a path to doing good on that front I feel good about.” Maybe you can walk me through how you got from there to where you are today. Firstly, I want to connect this back to how this relates to the work I was doing at GiveWell, and why this all falls under one theme. If we are on the cusp for this century of creating these advanced AI systems, then we could be looking at a future that's either very good or very bad. I think there are decent arguments that if we move forward without caution and we develop sloppily designed AI systems, they could end up with goals of their own. We would end up with a universe that contains very little that humans value or a galaxy that does or a world where very powerful technologies are used by ill-meaning governments to create a world that isn't very good. We could also end up with a world where we manage to eliminate a lot of forms of material scarcity and have a planet that's much better off than today's. A lot of what I ask is how can we help the most people possible per dollar spent? If you ask how we can help the most people possible per dollar spent, then funding some work to help shape that transition and make sure that we don't move forward too incautiously, and that we do increase the odds that we do get like a good future world instead of a bad future one, is helping a huge number of people per dollar spent. That's the motivation. You're quoting an argument I was having where we posted a transcript back in 2014–– a time that was part of my journey of getting here. I was talking to people who were saying, “Holden, you want to help a lot of people with your resources. You should be focused on this massive event that could be coming this century that very few people are paying attention to, and there might be a chance to make this go well or poorly for humanity.” So I was saying, “Gosh, like that sure is interesting.” And I did think it was interesting. That's why I was spending the time and having those conversations. But I said, “When I look at global poverty and global health, I see what I can do. I see the evidence. I see the actions I can take. I'm not seeing that with this stuff.” So what changed? I would say a good chunk of what changed is maybe like the most boring answer possible. I just kept at it. I was sitting there in 2014 saying, “Gosh, this is really interesting, but it's all a bit overwhelming. It's all a bit crazy. I don't know how I would even think about this. I don't know how I would come up with a risk from AI that I actually believed was a risk and could do something about today.” Now, I've just been thinking about this for a much longer time period. I do believe that most things you could say about the far future are very unreliable and not worth taking action on, but I think there are a few things one might say about what a transition to very powerful AI systems could look like. There are some things I'm willing to say would be bad if AI systems were poorly designed, had goals of their own, and ended up kind of running the world instead of humans. That seems bad. d look like. There are some things I'm willing to say would be bad if AI systems were poorly designed, had goals of their own, and ended up kind of running the world instead of humans. That seems bad. I am more familiar today than I was then with the research and the work people can do to make that less likely and the actions people can take to make that less likely–– so that's probably more than half the answer. But another thing that would be close to half the answer is that I think there have been big changes in the world of AI since then. 2014 was the beginning of what's sometimes called the “deep learning revolution”. Since then, we've basically seen these very computationally intensive (but fundamentally simple) AI systems achieve a lot of progress on lots of different unrelated tasks. It's not crazy to imagine that the current way people are developing AI systems, cutting-edge AI systems, could take us all the way to the kind of extremely powerful AI systems that automate roughly everything humans do to advance science and technology. It's not so wild to imagine that we could just keep on going with these systems, make them bigger, put more work into them, but basically stay on the same path and you could get there. If you imagine doing that, it becomes a little bit less daunting to imagine the risks that might come up and the things we could do about them. So I don't think it's necessarily the leading possibility, but it's enough to start thinking concretely about the problem.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.