Ilya Sutskever: evaluation

Transcript context

…I guess it’s not even that, but I do think that is an important part of it. The other big thing is that I can’t think of another discipline in human engineering and research where the end artifact was made safer mostly through just thinking about how to make it safe, as opposed to, why airplane crashes per mile are so much lower today than they were decades ago. Why is it so much harder to find a bug in Linux than it would have been decades ago? I think it’s mostly because these systems were deployed to the world. You noticed failures, those failures were corrected and the systems became more robust. I’m not sure why AGI and superhuman intelligence would be any different, especially given—and I hope we’re going to get to this—it seems like the harms of superintelligence are not just about having some malevolent paper clipper out there. But this is a really powerful thing and we don’t even know how to conceptualize how people interact with it, what people will do with it. Having gradual access to it seems like a better way to maybe spread out the impact of it and to help people prepare for it. Well I think on this point, even in the straight shot scenario, you would still do a gradual release of it, that’s how I would imagine it. Gradualism would be an inherent component of any plan. It’s just a question of what is the first thing that you get out of the door. That’s number one. Number two, I believe you have advocated for continual learning more than other people, and I actually think that this is an important and correct thing. Here is why. I’ll give you another example of how language affects thinking. In this case, it will be two words that have shaped everyone’s thinking, I maintain. First word: AGI. Second word: pre-training. Let me explain. The term AGI, why does this term exist? It’s a very particular term. Why does it exist? There’s a reason. The reason that the term AGI exists is, in my opinion, not so much because it’s a very important, essential descriptor of some end state of intelligence, but because it is a reaction to a different term that existed, and the term is narrow AI. If you go back to ancient history of gameplay and AI, of checkers AI, chess AI, computer games AI, everyone would say, look at this narrow intelligence. Sure, the chess AI can beat Kasparov, but it can’t do anything else. It is so narrow, artificial narrow intelligence. So in response, as a reaction to this, some people said, this is not good. It is so narrow. What we need is general AI, an AI that can just do all the things. That term just got a lot of traction. The second thing that got a lot of traction is pre-training, specifically the recipe of pre-training. I think the way people do RL now is maybe undoing the conceptual imprint of pre-training. But pre-training had this property. You do more pre-training and the model gets better at everything, more or less uniformly. General AI. Pre-training gives AGI. But the thing that happened with AGI and pre-training is that in some sense they overshot the target. If you think about the term “AGI”, especially in the context of pre-training, you will realize that a human being is not an AGI. Yes, there is definitely a foundation of skills, but a human being lacks a huge amount of knowledge. Instead, we rely on continual learning. So when you think about, “Okay, so let’s suppose that we achieve success and we produce some kind of safe superintelligence.” The question is, how do you define it? Where on the curve of continual learning is it going to be? I produce a superintelligent 15-year-old that’s very eager to go. They don’t know very much at all, a great student, very eager. You go and be a programmer, you go and be a doctor, go and learn. So you could imagine that the deployment itself will involve some kind of a learning trial-and-error period. It’s a process, as opposed to you dropping the finished thing. I see. You’re suggesting that the thing you’re pointing out with superintelligence is not some finished mind which knows how to do every single job in the economy. Because the way, say, the original OpenAI charter or whatever defines AGI is like, it can do every single job, every single thing a human can do. You’re proposing instead a mind which can learn to do every single job, and that is superintelligence.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.