Simon Willison: commitment

Transcript context

…I just wanted to give a shout out to Leon and his team. They just launched Garak, which is this tool for a security probing for LLMs, and it has a, an auto red teaming function. So that might be interesting to check out. But one, one thing that I was thinking about while Simon was talking and I read your blog post on prompt injection, and you know, on one hand we want to have more people exposing these vulnerabilities and maybe sharing their code for red teaming and their examples at the same time, that is helping people who want to abuse like these models and functions. So it's, it's, it's a tricky one. I'm curious how, how you're thinking about it. And I had a parting question as well, which is, what are we gonna ask next from OpenAI? What is missing and what would make our use of the technology of the API better? So this is a really interesting ethical question. The, it, it's like, All aspects of security service people about responsible disclosure of security vulnerabilities. The frustrating thing with prompt injection, we don't have a fix yet. So, you know, normally if you find a sequel injection hole in someone's website, you quietly tell them about it and they patch it, and then everything's fine with prompt injection. Seeing as there is no known way to fix these problems, I kind of feel like it's on us to make sure people understand it before they build systems that are vulnerable to them. So that, that's the approach I've been taking is just, just really trying to shake people and say, no, you can't just say, oh, we'll filter it out, it'll be fine. That doesn't work yet. You need to, sometimes you need to say, no, I cannot build the feature you're asking me to build because it can't be built securely. And that's really frustrating. I kind of hate that. Like I, I don't want to be the person who says there's a security hole, you need to stop. I want be the person who says there's security hole. Here's the fix for it, and now we can move on with our lives. But sadly we're, we're, we're not at that that point with it yet. I, I, I like, I 100% agree. Like, I, like it's been frus, it's been weird to me like it's gone from like supe suspicious to like frustrating, to like, just like curious, like, like this seems to be such a hard problem. Like it's a, like, there, there, they're like the, like I, I, I think what's going on really is that like, to solve this, you kind of had to re-engineer it to this message-based API because like they have now reserved tokens. They have tokens that only they know exist that they can insert it in as like quote marks. And they can have things like a system message. They can tune it in a way that it actually like, you know, like if you could peer into its brain and see like what circuit is it implementing? It's something that pays attention to the system message in the right way. That it like understands the difference between what, like the open AI customer told it our instructions and what the user told it. And like, I think that's progress in the right direction of like, that it's like a sensible, like target. Like it makes sense to me, like as an outsider of, of like, that's how you would like go about addressing this. But it just, like, it's a big change. And so like, I, I think they're moving towards it, but it like speaks to just like, what a hard problem this is. You know, because like, it, it is since May, I think since like preamble where the original like discoveries of it, you know, in May. And so when, when they put it in, you know, responsible disclosure and so it, it is just, it seems like it's just a deep like issue with how like, you know, the in instruction tuning or attention or something about this works that, that like, it's not trivially fixable and yeah, I think, you know, they, they're, they're making piece, you know, like piece by piece progress towards it. I think.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.