Nicholas Carlini: evaluation

Transcript context

…The fix was just don't show logits. Yeah, so the fix in particular was don't show log probs when you supply a logit bias. And what you don't show is the logit bias plus the log prob, which is like a very narrow thing. They sort of did the narrow thing to prevent this. Some people were unhappy, but like this is, you know, this is the nature of making, you can have a more useful system or a more secure system in many ways. I really like this example because for a very long time, nothing about GPT-4 would be at all different if the field, like the entire field of ever so much machine learning disappeared. Like everything to do with ever so examples, like all of like for the most part, like GPT-4 would exist identically. This is not true in other fields in system security. Like the way we design our processors today is fundamentally different because of the security attacks that we've had in the past. You know, the way we design databases, the way we design the internet is fundamentally different because of the way the attacks that we have. And what that means is it means that the attacks that we had were so compelling to the non-security people that they were willing to change and make their systems less useful in order to make the security better. In adversarial machine learning,we didn't have this. We didn't have attacks that were useful enough that you could show it to someone who actually designed a real system and they'd be willing to say, I am going to make my system less useful because the attack that you've presented to me is so compelling that I will break the functionality of my system. And this is one of the first cases I think that we were able to show this is someone, we had an attack that someone said, I agree with this attack is sufficiently bad that I will break utility in order to prevent this attack. And I would like to see more of these kinds of attacks, not because I want things to be worse, but because I want to be sure that we have exhausted the space of possible attacks so that it's not going to be the case that someone else comes up with a very bad thing that they're not going to disclose, sit on for a couple months, and then go and bang on everything and see what they can hit. And this is the hope of doing this research direction. I want to spell it out for people who are maybe not so specialized in this. Your attack could potentially steal the entire projection matrix.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.