Joscha Bach: prediction

Transcript context

…So that’s more like a stage four rationalist worry? Yes. If you are at stage five, you’re mostly worried that AI is not going to be enlightened fast enough, because you realize that the game is not so much about intelligence, but about agency, about the ability to control the future, and the identity is instrumental to this. If you are a human being, I think at some level, you ought to choose your own identity. You should not have somebody else pick the costume for you, and then wear it. But instead, you should be mindful about what you want to be in this world. I think if you are an agent that is fully malleable, that can provide its own source code like an AI might do at some point, then the identity that you will have is whatever you can be. In this way, the AI will maybe become everything like a planetary control system. If it does that, then if we want to coexist with it, it means that it’ll have to share purposes with us, so it cannot be a transactional relationship. We will not be able to use reinforcement learning with human feedback to hardwire its values into it. But this has to happen. It’s probably that it’s conscious, so it can relate to our own mode of existence, where an observer is observing itself in real-time, and within certain temporal frames. The other thing is that it probably needs to have some kind of transcendental orientation, building shared agency, in the same way as we do when we are able to enter with each other into non-transactional relationships. I find that’s something that, because the stage five is so rare, is missing in much of the discourse. I think that we need, in some sense, focus on how to formalize love, how to understand love, and how to build it into the machines that we are currently building, and that are about to become smarter than us. Well, I think this is a good opportunity to try to sneak up to the idea of enlightenment. You wrote a series of good tweets about consciousness, and panpsychism. Let’s break it down. First you say, I suspect the experience that leads to the panpsychism syndrome of some philosophers, and other consciousness enthusiasts represents the realization that we don’t end at the self, but share a resonant universe representation with every other observer coupled to the same universe. This actually, eventually leads us to a lot of interesting questions about AI, and AGI. But let’s start with this representation. What is this resonant universe representation, and what do you think? Do we share such a representation?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.