Eliezer Yudkowsky: evaluation

Transcript context

…Is there a level of global regulation at which you feel that the risk of everybody dying was less than 90%? It depends on the exit plan. How long does the equilibrium need to last? If we’ve got a crash program on augmenting human intelligence to the point where humans can solve alignment and managing the actual but not instantly automatically lethal risks of augmenting human intelligence. If we’ve got a crash program like that and we can think back, we only need 15 years of time and that 15 years of time may still be quite dear. 5 years should be a lot more manageable. The problem is that algorithms are continuing to improve. So you need to either shut down the journals reporting the AI results, or you need less and less and less computing power around. Even if you shut down all the journals people are going to be communicating with encrypted email lists about their bright ideas for improving AI. But if they don’t get to do their own giant training runs, the progress may slow down a bit. It still wouldn’t slow down forever. The algorithms just get better and better and the ceiling of compute has to get lower and lower and at some point you’re asking people to give up their home GPUs. At some point you’re being like — No more high speed computers. Then I start to worry that we never actually do get to the glorious transhumanist future and in this case, what was the point? Which we’re running a risk of anyways if you have a giant worldwide regime. (Unclear audio) Kind of digressing here. But my point is to get to like 90% chance of winning, which is pretty hard on any exit scheme, you want a fast exit scheme. You want to complete that exit scheme before the ceiling on compute is lowered too far. If your exit plan takes a long time, then you better shut down the academic AI journals and maybe you even have the Gestapo busting in people’s houses to accuse them of being underground AI researchers and I would really rather not live there and maybe even that doesn’t work. Let me know if this is inaccurate but I didn’t realize how much of the successful branch of decision tree relies on augmented humans being able to bring us to the finish line…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.