Gary Marcus: commitment

Transcript context

…So I I think that the process that I described earlier, that sort of recursive loop, I guess you could call it intelligence recursion, which if it goes fast enough is an intelligence explosion, That is not something you can research your way out of. You can't just, you know, write an 8 page paper and then we've solved it, you know, in the news tomorrow. It's we've out how to fully de risk some extremely fast moving process that we've never done before, and all of its unknown unknowns have been anticipated. Well, mean, we could politically I mean, it would take a lot of willpower, and it's probably not likely. But we could, you know, try to have a global treaty. Don't go there. Don't work on these kinds of things. Let's report it. If you do, da da da. Right? I mean you could at least imagine that kind of scenario. I don't see any technical way of coping necessarily with that set of problems right now. But if we were, I mean not just the 3 of us in the room, but if we as a society were convinced that let's say that was a red line. You suggested 1 red line, there are others. So we could decide as a society there are certain red lines, maybe recursive self improvement might be a reasonable 1 to consider, that we could say we won't cross any of the red lines. We will make treaties around it, and we just shouldn't do it until we have answers to them. So I think it would be worth states clarifying that. We don't want anybody doing that type of fully automated intelligence recursion because it'd be destabilizing. And now that doesn't necessarily immediately take the form of a treaty. So initially you have them exchanging words about that and articulating their preferences potentially explicitly or potentially in internal policy at CIA and things like that, and the other parties come to learn it through leaks or directly. And you eventually want to gain more confidence that nobody's trying to trigger something like this. And this this process is multiple stages. It may involve things even like skirm it requires a conversation, it may even require a skirmish for for people to think, okay, we need to do a a verification now. And then maybe you get some type of of of treaty, but you can still have various forms of coordination through deterrence without anything like a treaty, just as we've had strategic stability on multiple issues without treaties necessarily. So that's something that is potentially a later stage thing, but you have to have the conversation advance further.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.