Ilia Shumailov: prediction

Transcript context

…Yeah. I am extremely worried about this. And I'm less worried for industry because industry controls its supply chain. Like, everything is significantly better. But for an average consumer, have you heard about Block 4 j vulnerability? That kind of the thing that stormed the Internet. There was, like, hundreds of millions of compromises. It was basically the standard library that is used for logging in, basically, all of the Java applications that were running in the past. And at 1 point, people realized that when you write into the logs, the identity itself can be a remote identity. So you can basically say, there exists this class that is serialized somewhere on the Internet, and you, as a logging utility, if you find an identity to a remote remote class, you need to go, load the external code, deserialize this thing. You run this and exec it, basically, and then you know the identity of the thing that throws something into the lock. So what people found out is that they can inject those, like, remote code references that gets pulled inside and executed. And this thing opened a Pandora box because this low 4 j thing was everywhere, and you get arbitrary code execution on the box. And the basic primitive inside was it's a reference to external code that is loaded inside and just executes inside. And this caused a massive havoc all across the world, in all of our computer system. Honestly, if you try and read around on the number of compromises, we're talking about hundreds of millions of devices. Okay? Now we look at Hugging Face as a library, and you look at this wonderful flag called trust remote code. And what this thing does is that when you load the model, you know, like, you click use this model, use transformers. Inside, it gives you, like, a code snippet to load some model. Inside, it has this flag sometimes hard coded. What And this thing does is they say, oh, for some models, when you load them, you actually wanna load the latest the latest representation from an external machine. What this thing does is literally remote code, load it on your machine, execute it on your machine, load it on top of stuff. So same sort of thing we did back then. We're doing the same again. Today on Hugging Face, I don't know how many users there are. But if you're running your thing outside of a jail, if you're running your model outside of a sandbox, you are doing a very bad thing to yourself. And the other thing I have to say is there has been at least publicly 2 reported compromises of the CICD integration for PyTorch. On GitHub, there is, like, an automatic runner. Every time there is a build, they basically automatically do all the tests and stuff. Somebody broke into those runners. And when you break into these runners, you can change the build files themselves. So you can sort of whatever you want. You can change the code. We also had 2 2 instances, and when people were reporting I I encourage people to read through this. n change the build files themselves. So you can sort of whatever you want. You can change the code. We also had 2 2 instances, and when people were reporting I I encourage people to read through this. You can find references in the papers on supply chain. The other side couldn't figure out what was wrong with this. So it was at least, I think, timeline is half a year in 1 of the cases until they figured out and fixed stuff. And then there is also another thing that happened was somebody broke the PyTorch nightly build by playing around with the priorities of where the packages are loaded from. They noticed that there is 1 of the Torch packages that is loaded during the build phase that is not actually registered on the main package distribution platform. So they registered this, put malware inside of this. It got pulled into the standard PyTorch nightly build, and, apparently, they had a couple of thousand downloads of this. This is the norm today. I think we will have a lot more compromise to the point when they become more useful. And this is, like, public facing things. In industry, it's slightly different because industry actually controls all of this package management by themselves. They have proper dedicated teams looking at supply chains. But, like, I I think in the sort of, like, consumer space, no. It's it's it's actually very spooky. I don't even I don't even trust industry for this is why I wouldn't install Claude code on my personal machine. I'm like Don't do that. No way I'm gonna do that. Like, I'll I'll I'll let's get a VM. I'll put it on a VM. That's fine. I'm not putting it on my personal computer.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.