Sander Schulhoff: evaluation

Transcript context

…Please share. So the first thing that we need to understand is that the number of possible attacks against another LLM is equivalent to the number of possible prompts. Each possible prompt could be an attack. And for a model like GPT-5, the number of possible attacks is one followed by a million zeros. And to be clear, not a million attacks. A million has six zeros in it. We're saying one followed by one million zeros. That's so many zeros. That's more than a google worth of zeros. It's basically infinite. It's basically an infinite attack space. And so, when these guardrail providers say, "Hey," I mean, some of them say, "Hey, we catch everything." That's a complete lie, but most of them say, "Okay, we catch 99% of attacks." Okay. 99% of one followed by a million zeros, there's just so many attacks left. There's still basically infinite attacks left. And so, the number of attacks they're testing to get to that 99% figure is not statistically significant. It's also an incredibly difficult research problem to even have good measurements for adversarial robustness. And in fact, the best measurement you can do is an adaptive evaluation. And what that means is you take your defense, you take your model or your guardrail, and you build an attacker that can learn over time and improve its attacks. One example of adaptive attacks are humans. Humans are adaptive attackers because they test stuff out and they see what works and they're like, "Okay, this prompt doesn't work, but this prompt does. " And I've been working with people running AI red teaming competitions for quite a long time and will often include guardrails in the competition and the guardrails get broken very, very easily. And so, we actually, we just released a major research paper on this alongside OpenAI, Google DeepMind, and Anthropic that took a bunch of adaptive attacks. So these are like RL and search-based methods, and then also took human attackers and threw them all at all the state-of-the-art models, including GPT-5, all the state-of-the-art defenses. And we found that, first of all, humans break everything. A hundred percent of the defenses in maybe like 10 to 30 attempts. Somewhat interestingly, it takes the automated systems a couple orders of magnitude more attempts to be successful. And even then they're only, I don't know, maybe on average can be 90% of the situations. So human attackers are still the best, which is really interesting because a lot of people thought you could kind of completely automate this process. But anyways, we put a ton of guardrails in that event, in that competition, and they all got broken quite, quite easily. So another angle on the guardrails don't work. You can't really state you have 99% effectiveness because it's such a large number that you can never really get to that many attempts. And they can't prevent a meaningful amount of attacks because there's basically infinite attacks. 9% effectiveness because it's such a large number that you can never really get to that many attempts. And they can't prevent a meaningful amount of attacks because there's basically infinite attacks. But maybe a different way of measuring these guardrails is like, do they dissuade attackers? If you add a guardrail on your system, maybe it makes people less likely to attack. And I think this is not particularly true either, unfortunately, because at this point it's somewhat difficult to trick GPT-5. It's decently well-defended and adding a guardrail on top, if someone is determined enough to trick GPT-5, they're going to deal with that guardrail. No problem. No problem. So they don't dissuade attackers. Yeah, other things of particular concern. I know a number of people working at these companies, and I am permitted to say these things, which I will approximately say, but they tell me things like the testing we do is. They're fabricating statistics, and a lot of the times their models don't even work on non-English languages or something crazy like that, which is ridiculous because translating your attack to a different language is a very common attack pattern. And so, if it doesn't work in English, it's basically completely useless. So there's a lot of aggressive sales maybe and marketing being done, which is quite important. Another thing to consider if you're kind of on the fence and you're like, "Well, these guys are pretty trustworthy." I don't know, they seemed like they have a good system is the smartest artificial intelligence researchers in the world are working at Frontier Labs like OpenAI, Google, Anthropic. They can't solve this problem. They haven't been able to solve this problem in the last couple years of large language models being popular.This actually isn't even a new problem. Adversarial robustness has been a field for, oh gosh, I'll say like the last 20 to 50 years. I'm not exactly sure, but it's been around for a while, but only now is it in this kind of new form where, well, frankly, things are more potentially dangerous if the systems are tricked, especially with the agents. And so if the smartest AI researchers in the world can't solve this problem, why do you think some random enterprise who doesn't really even employ AI researchers can? It just doesn't add up. And another question you might ask yourself is, they applied their automated red teamer to your language models and found attacks that worked. What happens if they apply it to their own guardrail? Don't you think they'd find a lot of attacks that work? They would. They would. And anyone can go and do this. So that's the end of my guardrails don't work, Rant. Yeah, let me know if you have any questions about that.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.