Eliezer Yudkowsky: prediction

Transcript context

…So if there is a point at which we can get the public momentum to do some sort of stop, wouldn’t it be useful to exercise it when we get a GPT-6? And who knows what it’s capable of. Why do it now? Because allegedly, and we will see, people right now are able to appreciate that things are storming ahead a bit faster than the ability to ensure any sort of good outcome for them. And you could be like — “Ah, yes. We will play the galaxy-brained clever political move of trying to time when the popular support will be there.” But again, I heard rumors that people were actually completely open to the concept of let’s stop. So again, I’m just trying to say it. And it’s not clear to me what happens if we wait for GPT-5 to say it. I don’t actually know what GPT-5 is going to be like. It has been very hard to call the rate at which these systems acquire capability as they are trained to larger and larger sizes and more and more tokens. GPT-4 is a bit beyond in some ways where I thought this paradigm was going to scale. So I don’t actually know what happens if GPT-5 is built. And even if GPT-5 doesn’t end the world, which I agree is like more than 50% of where my probability mass lies, maybe that’s enough time for GPT-4.5 to get ensconced everywhere and in everything, and for it actually to be harder to call a stop, both politically and technically. There’s also the point that training algorithms keep improving. If we put a hard limit on the total computes and training runs right now, these systems would still get more capable over time as the algorithms improved and got more efficient. More oomph per floating point operation, and things would still improve, but slower. And if you start that process off at the GPT-5 level, where I don’t actually know how capable that is exactly, you may have a bunch less lifeline left before you get into dangerous territory. The concern is then that — there’s millions of GPUs out there in the world. The actors who would be willing to cooperate or who could even be identified in order to get the government to make them cooperate, would potentially be the ones that are most on the message. And so what you’re left with is a system where they stagnate for six months or a year or however long this lasts. And then what is the game plan? Is there some plan by which if we wait a few years, then alignment will be solved? Do we have some sort of timeline like that?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.