Daniel Kokotajlo: prediction

Transcript context

…So, we were just getting lunch with a prominent Washington, D.C. political journalist, and he was making the point that when he talks to these congresspeople, when he talks to political leaders, none of them are at all awake to the possibility even of stronger AI systems, let alone AGI, let alone superhuman intelligence. I think a lot of your forecast relies on, at some point, not only the US President, but also Xi Jinping, waking up to the possibility of a super intelligence and the stakes involved there. Why think that even when you show Trump the remote worker demo, he’s going to be like, “oh, and therefore in 2028, there will be a super intelligence. Whoever controls that will be God emperor forever”. Maybe not that extreme, but you see what I’m saying. Why wouldn’t he just be like, “there’ll be a stronger remote worker in 2029, a better remote worker in 2031”? Well, to be clear, we are uncertain about this, but in our story, we depict this sort of intense wake up happening over the course of 2027, mostly concurrently with the AI companies automating all of their R&D internally and having these fully autonomous agents that are amazing autonomous hackers and stuff like that, but then also actually doing all the research. And part of why we think this wakeup happens is because the company deliberately decides to wake up the president. You could imagine running the scenario with that not happening. You can imagine the companies trying to sort of keep the president in the dark. I do think that they could do that. I think that if they didn’t want the President to wake up to what’s going on, they might be able to achieve that. Strategically though, that would be quite risky for them. Because if they keep the President in the dark about the fact that they’re building superintelligence and that they’re actually completely automated their R&D and it’s getting superhuman across the board, and then if the President finds out anyway somehow, perhaps because of a whistleblower, he might be very upset at them and he might crack down really hard and just actually execute on all the threats and nationalize them and blah, blah, blah. They want him on their side. And to get him on their side, they have to make sure he’s not surprised by any of these crazy developments. And also, if they do get him on their side, they might be able to actually go faster. They might be able to get a lot of red tape waived and stuff like that. And so we made the guess that early in 2027, the company would basically be like, ‘We are going to deliberately wake up the president and scare the president with all of these demos of crazy stuff that could happen, and then use that to lobby the President to help us go faster and to cut red tape and to maybe slow down our competitors a little bit and so forth. ’ We also are pretty uncertain how much opposition there’s going to be from civil society and how much trouble that’s going to cause for the companies. So people who are worried about job loss, people who are worried about art, copyright, things like that, maybe enough of a bloc that AI becomes extremely politically unpopular. I think we have OpenBrain, our fictional company’s net approval ratings getting down to minus 40, minus 50 sometime around this point. So I think they’re also worried that if the President isn’t completely on their side, then they might get some laws targeting them, or they may just need the president on their side to swat down other people who are trying to make laws targeting them. And the way to get the President on their side is to really play up the national security implications.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.