Daniel Kokotajlo: uncertainty

Transcript context

…It’s alignment problems all the way down. Yeah. And so, for example, I’m not a lawyer, I don’t know the details about how this would work out, but I have a sort of high-level strategic picture of the fight between the White House and the CEO. And the strategic picture is basically the White House can sort of threaten, “here’s all these orders I could make, Defense Production Act, blah, blah, blah. I could do all this terrible stuff to you and basically disempower you and take control”. And then the CEO can threaten back and be like, “here’s how we would fight it in the courts, here’s how we would fight it in the public. Here’s all this stuff we would do”. And after then they both do their posturing with all their threats, then they’re like, “okay, how about we have a contract that instead of executing on all of our threats and having all these crazy fights in public, we’ll just come to a deal and then have a military contract that sets out who gets to call what shots in the company”. And so that’s what we depict happening is that they don’t blow up into this huge power struggle publicly, instead they negotiate and come to some sort of deal where they basically share power. And there is this oversight committee that has some members appointed by the President and also the CEO and his people. And that committee votes on high level questions like “what goals should we put into the superintelligences?”. So, we were just getting lunch with a prominent Washington, D.C. political journalist, and he was making the point that when he talks to these congresspeople, when he talks to political leaders, none of them are at all awake to the possibility even of stronger AI systems, let alone AGI, let alone superhuman intelligence. I think a lot of your forecast relies on, at some point, not only the US President, but also Xi Jinping, waking up to the possibility of a super intelligence and the stakes involved there. Why think that even when you show Trump the remote worker demo, he’s going to be like, “oh, and therefore in 2028, there will be a super intelligence. Whoever controls that will be God emperor forever”. Maybe not that extreme, but you see what I’m saying. Why wouldn’t he just be like, “there’ll be a stronger remote worker in 2029, a better remote worker in 2031”?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.