Sander Schulhoff: evaluation

Transcript context

…Okay. Extremely helpful. And as people have been listening to this, I imagine they're all thinking, why can't you just add some code in front of this thing of just like, "Okay, if it's telling someone to write a bomb, don't let them do that. If it's trying to change our database, stop it from doing that." And that's this whole space of guardrails is companies are building these... It's probably AI-powered plus some kind of logic that they write to help catch all these things. This ServiceNow example, actually, interestingly, ServiceNow has a prompt injection protection feature and it was enabled as this person was trying to hack it and they got through. So that's a really good example of, okay, this is awesome. Obviously a great idea. Before we get to just how these companies work with enterprises and just the problems with this sort of thing, there's a term that you believe is really important for people to understand adversarial robustness. Explain what that means. Yeah. Adversarial robustness. Yeah. So this refers to how well models or systems... ... refers to how well models or systems can defend themselves against attacks. And this term is usually just applied to models themselves, so just large language models themselves. But if you have one of those like guardrail, then LLM, then another guardrail system, you can also use it to describe the defensibility of that term. And so, if 99% of attacks are blocked, I can say my system is like 99% adversarially robust. You'd never actually say this in practice because it's very difficult to estimate adversarial robustness because the search space here is massive, which we'll talk about soon. But it just means how well-defended a system is. Okay. So this is kind of the way that these companies measure their success, the impact they're having on your AI product, how robust and how good your AI system is a stopping bad stuff.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.