Holden Karnofsky: belief

Transcript context

…My story is that I originally co-founded an organization called GiveWell that helps people decide where to give as effectively as possible. While I’m no longer as active as I once was there, I'm on its board. It's a website called GiveWell.org that makes good recommendations about where to give to charity to help a lot of people. As we were working at GiveWell, we met Cari Tuna and Dustin Moskovitz. Dustin is the co-founder of Facebook and Asana and we started a project that became Open Philanthropy to try to help them give away their large fortune and help as many people as possible. So I've spent my career looking for ways to do as much good as possible with a dollar, an hour, or basically whatever resources you have (especially with money). I've developed this professional specialization in looking for ideas that are underappreciated, underrated, and tremendously important because a lot of the time that's where I think you can find what you might call an “outsized return on investment.” There are opportunities to spend money and get an enormous impact because you're doing something very important that’s being ignored by others. So it's through that kind of professional specialization that I've actively looked for interesting ideas that are not getting enough attention. Then I encountered the Effective Altruist Community, which is a community of people basically built around the idea of doing as much good as you can. It's through that community that I encountered the idea of the most important century. It's not my idea at all, I reached this conclusion with the help and input of a lot of people. The basic idea is that if we developed the right kind of AI systems this century (and that looks reasonably likely), this could make this century the most important of all time for humanity. So now let’s talk about the basic mechanics of why that might be or how you might think about that. One thing is that if you look back at all of economic history ( the rate at which the world economy has grown), you see acceleration. You see that it's growing a lot faster today than it ever was. One theory of why that might be or one way of thinking about it through the lens of basic economic growth theory is that in normal circumstances, you can imagine a feedback loop where you have people coming up with ideas, and then the ideas lead to greater productivity and more resources. When you have more resources, you can also have more people, and then those people have more ideas. So you get this feedback loop that goes people, ideas, resources, people, ideas, resources. If you’re starting a couple of hundred years ago and you run a feedback loop like that, standard economic theory says you'll get accelerating growth. You'll get a rate of economic growth that goes faster and faster. ing a couple of hundred years ago and you run a feedback loop like that, standard economic theory says you'll get accelerating growth. You'll get a rate of economic growth that goes faster and faster. Basically, if you take the story of our economy to date and you plot it on a chart and do the kind of simplest thing you can to project it forward, you project that it will go that our economy will reach an infinite growth rate this century. The reason that I currently don't think that's a great thing to expect by default is that one of the steps of that feedback loop broke a couple hundred years ago. So it goes more people, more ideas, more resources, more people, more ideas, more resources. But, a couple hundred years ago, people stopped having more children when they had more resources. They got richer instead of more populous. This is all discussed on the Most Important Century page on my blog, Cold Takes. What happens right now is that when we have more ideas and we have more resources, we don't end up with more people as a result. We don't have that same accelerating feedback loop. If you had AI systems that could do all the things humans do to advance science and technology (meaning the AI systems could fill in that “more ideas” part of the loop), then you could get that feedback loop back. You could get sort of this unbounded, heavily accelerating, explosive growth in science and technology. That's the basic dynamic at the heart of it and a way of putting it that's trying to use familiar concepts from economic growth theory. Another way of putting it might just be, “Gosh, if we had AI systems that could do everything humans do to advance science and technology, that would be insane.” What if we were to take the things that humans do to create new technologies that have transformed the planet so radically and we were able to completely automate them so that every computer we have is potentially another mind working on advancing technology? So either way, when you think about it, you could imagine the world changing incredibly quickly and incredibly dramatically. I argue in the Most Important Century series that it looks reasonably likely, in my opinion, more than 50-50, that this century will see AI systems that can do all of the key tasks that humans do to advance science and technology. If that happens, we'll see explosive progress in science and technology. The world will quickly become extremely different from how it is today. You might think of it as if there was thousands of years of changes packed into a much shorter time period. If that happens, then I argue that you could end up in a deeply unfamiliar future. I give one example of what that might look like using this hypothetical technology idea called digital people. That would be sort of people that live in virtual environments that are kind of simulated, but also realistic and exactly like us. When you picture that kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science and technology. kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science and technology. We could basically find most of what there is to find and end up with a civilization that expands well beyond this planet, has a lot of control over the environment, is very stable for very long periods of time, and looks sort of post-human in a lot of relevant ways. If you think that, then this is basically our last chance to shape how this happens. The most important century hypothesis in a nutshell is that if we develop AI that can do all the things humans do to advance science and technology, we could very quickly reach a very futuristic world that’s very different from today's. It could be a very stable, very large world, and this is our last chance to shape it.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.