Ilia Shumailov: belief

Transcript context

…are you gonna punish? What are you gonna do? Well, we'll just delete that agent and then version 2 will just do something else wrong. Right? So that's part of the problem too, right, is that there isn't the normal consequences. Right? The normal consequence chain doesn't really apply anymore. Yeah. Yeah. It's it's hard. I I think we really need to change our thinking and threat modeling because these agents, and obviously, they don't widely exist yet. This is a thing that is coming. Clearly, there is some benefit and early examples of things where they clearly made things better. But they're coming, I'm I'm pretty sure. And when they do come, expect that your insider threats, your corporate espionage things will go through the roof. Because the fundamentals of security do not change. Right? So, like, if an executive agent sends a financial agent a request to provide some financial interactions with some third party company and says, please send it over to someone else, what how should the financial agent know they're not supposed to do this? Right. They don't have enough context. They are not supposed to have enough context. So and and usually, when we talk about social engineering security, like, there is a a wide variety of things that, like, that describe why human systems fail, and it's gonna be a similar sort of thing because many of those problems, like, deputy problem is the formal name of this. Right? They they will exist. It's it's unlikely for them to disappear. Mhmm. Did you see that anthropic paper? What was it called? Agenic misalignment Mhmm. Where they I mean, you you better set this up better than me, but, know, they they set up this kind of contrived scenario, and the the AI tried to blackmail someone because they said that they were having an the boss was was having an affair or something like that. The AI didn't wanna be switched off. And it's just absolutely crazy what what these things do.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.