Eliezer Yudkowsky: prediction

Transcript context

…Because it’s not just a question of the technical feasibility of can you build a thing that applies its general intelligence narrowly to the neuroscience of augmenting humans? One, I feel like that is probably over 1% technical feasibility, but the world that we are in is so far from doing that, from trying the way that it could actually work. Like not the the try where — “Oh, you know. We'd like to do a bunch of RLHF to try to have a thing spit out output about this thing, but not about that thing” and no, not that. 1% that humanity could do that if it tried and tried in just the right direction as far as I can perceive angles in this space. Yeah, I’m over 1% on that. I am not very high on us doing it. Maybe I will be wrong. Maybe the Time article I wrote saying shut it all down gets picked up. And there are very serious conversations. And the very serious conversations are actually effective in shutting down the headlong plunge. And there is a narrow exception carved out for the kind of narrow application of trying to build an artificial general intelligence that applies its intelligence narrowly and to the problem of augmenting humans. And that, I think, might be a harder sell to the world than just shut it all down. They could shut it all down and then not do the things that they would need to do to have an exit strategy. I feel like even if you told me that they went for shut it all down I would expect them to have no exit strategy until the world ended anyways. But perhaps I underestimate them. Maybe there’s a will in humanity to do something else which is not that. And if there really were yeah, I think I’m even over 10% that would be a technically feasible path if they looked in just the right direction. But I am not over 50% on them actually doing the shut it all down. If they do that, I am then not over 50% on (unclear) them really having an exit strategy. Then from there you have to go in at sufficiently the right angle to materialize the technical chances and not do it in the way that just ends up a suicide, or if you’re lucky, gives you the clear warning signs and then people actually pay attention to those instead of just optimizing away the warning signs. And I don’t want to make this sound like the multiple stage fallacy of — “Oh more than one thing has to happen therefore the resulting thing can never happen.” Which super clear case in point of why you cannot prove anything will not happen this way. Nate Silver arguing that Trump needed to get through six stages to become the Republican presidential candidate each of which was less than half probability and therefore he had less than 1/64th chance of becoming the Republican candidate, not winning. You can’t just break things down into stages and then say therefore. The probability is zero. You can break down anything into stages. But even so, you’re asking me like — Isn’t over 1% that it’s possible? I’m like — yeah, possibly even over 10% . d then say therefore. The probability is zero. You can break down anything into stages. But even so, you’re asking me like — Isn’t over 1% that it’s possible? I’m like — yeah, possibly even over 10% . The reason why I tell people — “Yeah, don’t put your hope in the future, you’re probably dead”, is that the existence of this technical array of hope, if you do just the right things, is not the same as expecting that the world reshapes itself to permit that to be done without destroying the world in the meanwhile. I expect things to continue on largely as they have. And what distinguishes that from despair is that at the moment people were telling me, — “No, no. If you go outside the tech industry, people will actually listen.” I’m like — “All right, let’s try that. Let’s write the Time article. Let’s jump on that. It will lack dignity not to try.” but that’s not the same as expecting, as being like — “Oh yeah, I’m over 50%, they’re totally going to do it. That Time article is totally going to take off.” I’m not currently not over 50% on that. You said any one of these things could mean, and yet even if this thing is technically feasible, that doesn’t mean the world’s going to do it. We are presently quite far from the world being on that trajectory or of doing the things that would needed to create time to pay the alignment tax to do it. Maybe the one thing I would dispute is how many things need to go right from the world as a whole for any one of these paths to succeed. Which goes into the fourth point, which is that maybe the sort of universal prior over all the drives that an AI could have is just the wrong way to think about it.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.