Dwarkesh Patel: uncertainty

Transcript context

…Here’s one reason why I liked “AI that cares for sentient life”. We can debate on whether it’s good or bad. But if the first N of these dramatic systems do care for, love, humanity or something, care for sentient life, obviously this also needs to be achieved. This needs to be achieved. So if this is achieved by the first N of those systems, then I can see it go well, at least for quite some time. Then there is the question of what happens in the long run. How do you achieve a long-run equilibrium? I think that there, there is an answer as well. I don’t like this answer, but it needs to be considered. In the long run, you might say, “Okay, if you have a world where powerful AIs exist, in the short term, you could say you have universal high income. You have universal high income and we’re all doing well.” But what do the Buddhists say? “Change is the only constant.” Things change. There is some kind of government, political structure thing, and it changes because these things have a shelf life. Some new government thing comes up and it functions, and then after some time it stops functioning. That’s something that we see happening all the time. So I think for the long-run equilibrium, one approach is that you could say maybe every person will have an AI that will do their bidding, and that’s good. If that could be maintained indefinitely, that’s true. But the downside with that is then the AI goes and earns money for the person and advocates for their needs in the political sphere, and maybe then writes a little report saying, “Okay, here’s what I’ve done, here’s the situation,” and the person says, “Great, keep it up.” But the person is no longer a participant. Then you can say that’s a precarious place to be in. I’m going to preface by saying I don’t like this solution, but it is a solution. The solution is if people become part-AI with some kind of Neuralink++. Because what will happen as a result is that now the AI understands something, and we understand it too, because now the understanding is transmitted wholesale. So now if the AI is in some situation, you are involved in that situation yourself fully. I think this is the answer to the equilibrium. I wonder if the fact that emotions which were developed millions—or in many cases, billions—of years ago in a totally different environment are still guiding our actions so strongly is an example of alignment success. To spell out what I mean—I don’t know whether it’s more accurate to call it a value function or reward function—but the brainstem has a directive where it’s saying, “Mate with somebody who’s more successful. ” The cortex is the part that understands what success means in the modern context. But the brainstem is able to align the cortex and say, “However you recognize success to be—and I’m not smart enough to understand what that is— you’re still going to pursue this directive.” I think there’s a more general point. I think it’s actually really mysterious how evolution encodes high-level desires. It’s pretty easy to understand how evolution would endow us with the desire for food that smells good because smell is a chemical, so just pursue that chemical. It’s very easy to imagine evolution doing that thing. But evolution also has endowed us with all these social desires. We really care about being seen positively by society. We care about being in good standing. All these social intuitions that we have, I feel strongly that they’re baked in. I don’t know how evolution did it because it’s a high-level concept that’s represented in the brain. Let’s say you care about some social thing, it’s not a low-level signal like smell. It’s not something for which there is a sensor. The brain needs to do a lot of processing to piece together lots of bits of information to understand what’s going on socially. Somehow evolution said, “That’s what you should care about.” How did it do it? It did it quickly, too. All these sophisticated social things that we care about, I think they evolved pretty recently. Evolution had an easy time hard-coding this high-level desire. I’m unaware of a good hypothesis for how it’s done. I had some ideas I was kicking around, but none of them are satisfying.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.