Richard Sutton: prediction

Transcript context

…I agree with all four of those arguments and the implication. I also agree that succession contains a wide variety of possible futures. Curious to get more thoughts on that. I do encourage people to think positively about it. First of all, it’s something we humans have always tried to do for thousands of years, try to understand ourselves, trying to make ourselves think better, just understanding ourselves. This is a great success for science, humanities. We’re finding out what this essential part of humanness is, what it means to be intelligent. Then what I usually say is that this is all human-centric. But if we step aside from being a human and just take the point of view of the universe, this is I think a major stage in the universe, a major transition, a transition from replicators. We humans and animals, plants, we’re all replicators. That gives us some strengths and some limitations. We’re entering the age of design because our AIs are designed. Our physical objects are designed, our buildings are designed, our technology is designed. We’re designing AIs now, things that can be intelligent themselves and that are themselves capable of design. This is a key step in the world and in the universe. It’s the transition from the world in which most of the interesting things that are, are replicated. Replicated means you can make copies of them, but you don’t really understand them. Right now we can make more intelligent beings, more children, but we don’t really understand how intelligence works. Whereas we’re reaching now to having designed intelligence, intelligence that we do understand how it works. Therefore we can change it in different ways and at different speeds than otherwise. In our future, they may not be replicated at all. We may just design AIs, and those AIs will design other AIs, and everything will be done by design and construction rather than by replication. I mark this as one of the four great stages of the universe. First there’s dust, it ends with stars. Stars make planets. The planets can give rise to life. Now we’re giving rise to designed entities. I think we should be proud that we are giving rise to this great transition in the universe. It’s an interesting thing. Should we consider them part of humanity or different from humanity? It’s our choice. It’s our choice whether we should say, “Oh, they are our offspring and we should be proud of them and we should celebrate their achievements.”Or we could say, “Oh no, they’re not us and we should be horrified.” It’s interesting that it feels to me like a choice. Yet it’s such a strongly held thing that, how could it be a choice? I like these sort of contradictory implications of thought. It is interesting to consider if we are just designing another generation of humans. Maybe design is the wrong word. But we know a future generation of humans is going to come up. Forget about AI. We just know in the long run, humanity will be more capable and more numerous, maybe more intelligent. How do we feel about that? I do think there are potential worlds with future humans that we would be quite concerned about.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.