Daniel Kokotajlo: belief

Transcript context

…We expect that as the AI labs become more capable, they tell the government about this because they want government contracts, they want government support. Eventually it reaches the point where the government is extremely impressed. In our scenario, that starts with cyber warfare, the government sees that these AIs are now as capable as the best human hackers, but can be deployed at humongous scale. So they become extremely interested and they discuss nationalizing the AI companies. In our scenario, they never quite get all the way, but they’re gradually bringing them closer and closer to the government orbit. Part of what they want is security, because they know that if China steals some of this and they get these superhuman hackers, and part of what they want is just knowledge and control over what’s going on. So through our scenario, that process is getting further and further along, until by the time that the government wakes up to the possibility of superintelligence, they’re already pretty cozy with the AI companies. They already understand that superintelligence is kind of the key to power in the future. And so they are starting to integrate some of the national security state with some of the leadership of the AI companies so that these AIs are programmed to follow the commands of important people rather than just doing things on their own. If I may add to that. So by the government, I think what Scott meant is the executive branch, especially the White House. So we are depicting a sort of information asymmetry where the judiciary is out of the loop and the Congress is out of the loop and it’s mostly the executive branch that’s involved. Two, we’re not depicting government ultimately ending up in total control at the end. We’re thinking that there’s an information asymmetry between the CEOs of these companies and the President and they… It’s alignment problems all the way down.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.