Sander Schulhoff: evaluation

Transcript context

…Okay. To further secure people before we get into how people are trying to solve this problem, clearly something that isn't intended, all these behaviors. It's one thing for ChatGPT to tell you, "Here's how to build a bomb." That's bad. We don't want that. But as these things start to have control over the world, as agents become more populous, and as robots become a part of our daily lives, this becomes much more dangerous and significant. Maybe chat about that impact there that we might be seeing. I think you gave the perfect example with ServiceNow, and that's the reason that this stuff is so important to talk about right now because with chatbots, as you said, very limited damage outcomes that could occur, assuming they don't invent a new bioweapon or something like that. But with agents, there's all types of bad stuff that can happen. And if you deploy improperly secured, improperly data-permissioned agents, people can trick those things into doing whatever, which might leak your user's data and might cost your company or your user's money, all sorts of real world damages there. And we're going into robotics too, where they're deploying VLM, visual language model, powered robots into the world and these things can get prompt injected. And if you're walking down the street next to some robot, you don't want somebody else to say something to it that tricks it into punching you in the face, but that can happen. We've already seen people jailbreaking LM powered robotic systems, so that's going to be another big problem. Okay. So we're going to go on an arc. The next phase of this arc is maybe some good news as a bunch of companies have sprung up to solve this problem. Clearly this is bad. Nobody wants this. People want this solved. All the foundational models care about this and are trying to stop this. AI products want to avoid this like ServiceNow does not want their agents to be updating their database. So a lot of companies spring up to solve these problems. Talk about this industry.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.