Zvi Mowshowitz: prediction

Transcript context

…Let's I I would be happy to talk shop all day, but let's maybe zoom out from the parochial problems of the AI analysts and tackle the problems of the AI developers and the regulators. No need to recap events, but I guess I'd start with just this very big question of how should we understand what we've recently seen? And one way I've been thinking about it myself is we're somewhere and presumably, the investigation will give us a lot more clarity on exactly where. But I kinda think we're somewhere on a spectrum with these incidents from real recklessness where it was like, did you not have any monitoring going on on the one hand to on the other hand, like, the less reckless they are, the more scary the fundamentals are. Right? If you had great monitoring and this still happened, then, like, holy shit. That's really wild. Given everything you know right now, do you like that mental model? And where would you put us on that spectrum? Or you can obviously redefine give me your own spectrum. You'd simultaneously be horrified by and grateful for these forms of complete recklessness and incompetence on the infrastructure and supervision sides by these companies. On the one hand, this is a horrible situation they absolutely have to fix. And, like, we're so fucked if we don't fix it. But on other hand, that can be fixed. And by not fixing it, we get to see these things while they're relatively harmless, while they are relatively preventable, while they are in their easy platonic forms and, like, can be appreciated. On the flip side, that that gives people the excuse of, oh, these people are just incompetent. And that can cause people to dismiss the underlying situation. So it it does work both ways. To me, like, you know, we we we've seen failure on every level. It's like, I called it a total less wrong victory in the sense that everything is going the way you predicted and a total less wrong defeat in the sense that everything is going the way you predicted. Where, like, we did predict. You know, the Elias has the law of earlier failure, which is that, you know, plan will fail at a much earlier point for much stupider and more preventable reasons than you thought it would fail even if you thought the plan would definitely fail and had good reasons why it would definitely fail. But you can't if you would explain to people two years ago, OpenAI's models are gonna be misaligned, and they're gonna go out there and they're gonna hack major websites because OpenAI will just not care if their sandboxes are misconfig are not con are not strong enough to hold the AI. The AI will break out repeatedly. They will notice this. They'll be worried about this, but they will just leave the sandbox there for the AI to break out of while the safeguards are down and they just don't look at it for an entire week. People would say, that's stupid. Nobody is that incompetent. That would never happen. And your scenario makes no sense, and they would use this to then dismiss these stupid doomer concerns or whatever. Whatever. Because, like, obviously, like, people will just but people won't just. Right? People will never be just in this sense. Right? You know, people have never just anything, and they're not gonna start now. And we need these displays of utter incompetence and dirtiness in a general sense. And, like, one of things I've been hammering hammering is if your plan cannot survive the real world level of dirtiness and incompetence and ordinary human error, then your plan is insufficiently foolproof because of all the fools, and it will definitely fail. Even if your plan would have succeeded if we were not fools, we were competent, and we were responsible. So my position basically is, you know, I have the tweet I was handling right before we started this was Dean Ball's tweet about with even moderate prudence, things will probably go extraordinarily well. I don't think this is true. ou know, I have the tweet I was handling right before we started this was Dean Ball's tweet about with even moderate prudence, things will probably go extraordinarily well. I don't think this is true. I think we need more than moderate prudence to have good odds of success. And I think even with a lot of prudence, we would have a large odds of things not going well even if we did everything basically right, short of, you know, types of international and full cooperation that, you know, are reasonably unprecedented in many ways and, like, are not are not are nothing like moderate prudence. Right? Like, are are well beyond that. And that's just sort of a fact of the world we have. We have to live with that. We have to operate with that. But we also aren't gonna get modern recruitment by default. Like, we're gonna get complete incompetence. That's what we've been getting so far. Right? We've we've got a White House that, like, takes meetings with Bessette and Lutnick, who have no idea what AI, how how modern no one's work. Like, they're econ guys. Right? Even if I assume that they are well meeting, hardworking, competent guys for the positions in which they were nominated and confirmed and serve, This is just a completely different set of problems that they don't know how to handle. They don't understand them. And they have way too many other things going on to then drop everything they're doing and take six months to learn. Obviously, they couldn't possibly. So, like, what hope do you have? Meanwhile, the the AI companies that are built on the most paranoia, the most understanding of the problem, the most appreciation for how dangerous these things are, where all of the engineers actually expect superintelligence and and understand that there things are accelerating and things are dangerous, they still lower the cybersecurity safeguards in their untested new advanced model and then go away for a week. Like, literally, that that part did boggle my mind. Right? The the part where the AIs have these classic alignment failures. Right? Like, this is paper clip maximizer style failings by these AIs. These are standard. We gave you a goal and you pursued the goal even though it is completely obvious to you that the developer wouldn't want you to do that. The user wouldn't want you to do that. The consequences for you as an AI are not going to be good. The consequences for the world are not going to be good.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.