Speaker unverified: commitment

Transcript context

…From inside the model, we widen the lens to the whole civilization. David Duvenot is a machine learning professor at Toronto who spent time on the technical staff at Anthropic and co-wrote the paper Gradual Disempowerment. His concern isn't a rogue AI. It's that even if alignment basically works, a long series of small, sensible decisions to hand things over could still leave humanity, collectively, out of the driver's seat. He starts with monkeys. I think a lot of people gestured towards this when they said, oh, I'm worried about concentration of power, or I'm worried about not being the most competitive species on Earth. Like very intuitive arguments about, hey, we're not going to be on top because we're not going to be competitive. And then I think there was sort of this sophisticated, or at least response to this, which was, no, it'll be fine because we'll have AI to help us. Like when I talk to people at the big, at the major labs, including, well, they would say something like, well, sure, like in a normal world, we might not be fine. But hey, some of them, some people just have an intuition that the government is going to step in and make sure everyone's fine. Other people have an intuition that if everybody has an AI advisor helping them solve coordination problems, that will be well represented in whatever power struggle occurs. And my basic rebuttal that is like, well, the optimization process of like civilization or competition or techno capital or whatever, like you, people have different ways of describing agency to the emergence sort of allocation of resources towards growth that happens is just going to always be working against us and will always be fighting the current because we will be dragged on growth. We will have AIs that represent us, but these emerging growth centers that might not be allowed to humans will also have AIs helping them solve coordination problems and maybe crush scent. Yeah, sorry, what do you mean by growth? Like as in economic growth or? h centers that might not be allowed to humans will also have AIs helping them solve coordination problems and maybe crush scent. Yeah, sorry, what do you mean by growth? Like as in economic growth or? I basically mean economic growth. I mean, it's kind of funny because We only really have a good vocabulary for economic growth, but population growth is almost the same thing. And especially when you have AIs that are both sort of population and capital, it kind of merges. I mean, and then again, like think of like factories and power plants and robots, like anything that kind of can affect matter and cause there to be more growth. Let me just to stop your one intuition that a lot of people have is like, but surely human desires or consumption or something is always going to be what matters. And surely any corporation or government that we build is going to be like ultimately sort of being, the shots will be being called by human. And it just makes me think of like some monkeys and they're like trading bananas amongst each other and they see humans start to like build their city and they're like, oh wow, like, you know, we could probably trade with those humans and get rich. But of course, ultimately what matters is the banana monkey economy. And it might be hard to measure GDP if we don't count the human activity, but ultimately it's going to be the monkeys that are calling the shots because they're going to need to trade with us for bananas. And it's just like, people just don't understand that they might be irrelevant someday as like consumers or producers. And there can just be complete, it's just not that hard for them to be self-contained sources of growth that just don't have to answer to any particular human desire. And I mean, government is the classic one, right? Like The North Koreans never were like, hey, let's all get together and make a horrible system of government that oppressed us. And I don't think that like the Kims either were like, oh yeah, let's make this like horrible equilibrium. It just kind of happened. Same with like the USSR, same with like all kinds of states throughout history. It's just so easy for you to like to accidentally build a layer of agency on top of you that doesn't actually care about you. It cares about growth and power just emergently.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.