Eliezer Yudkowsky: evaluation

Transcript context

…Really? Someone might say — Listen Eliezer, according to the CEO of the company who is leading the AI race, he tweeted something that you’ve done the most to accelerate AI or something which was presumably the opposite of your goals. And it seems like other people did see that these sort of language models would scale in the way that they have scaled. Given that you didn’t see that coming and given that in some sense, according to some people, your actions have had the opposite impact that you intended. What is the track record by which the rest of the world can come to the conclusions that you have come to? These are two different questions. One is the question of who predicted that language models would scale? If they put it down in writing and if they said not just this loss function will go down, but also which capabilities will appear as that happens, then that would be quite interesting. That would be a successful scientific prediction. If they then came forth and said — this is the model that I used, this is what I predict about alignment. We could have an interesting fight about that. Second, there’s the point that if you try to rouse your planet to give it any sense that it is in peril. There are the idiot disaster monkeys who are like — “Ooh. Ooh. If this is dangerous, it must be powerful. Right? I’m going to be the first to grab the poison banana.” And what is one supposed to do? Should one remain silent? Should one let everyone walk directly into the whirling razor blades? If you sent me back in time, I’m not sure I could win this, but maybe I would have some notion of like if you calculate the message in exactly this way, then this group will not take away this message and you will be able to get this group of people to research on it without having this other group of people decide that it’s excitingly dangerous, and they want to rush forward on it. I’m not that smart. I’m not that wise. But what you are pointing to there is not a failure of ability to make predictions about AI. It’s that if you try to call attention to a danger and not just have everybody just have your whole planet walk directly into the whirling razor blades carefree, no idea what’s coming to them, maybe yeah that speeds up timelines. Maybe then people are like — “Ooh. Ooh. Exciting. Exciting. I want to build it. I want to build it. OOH, exciting. It has to be in my hands. I have to be the one to manage this danger. I’m going to run out and build it.” Like —Oh no. If we don’t invest in this company, who knows what investors they’ll have instead that will demand that they move fast because of the profit motive. Then of course, they just move fast fucking anyways. And yeah, if you sent me back in time, maybe I’d have a third option. But it seems to me that in terms of what one person can realistically manage, in terms of not being able to exactly craft a message with perfect hindsight that will reach some people and not others, at that point, you might as well just be like — Yeah, just invest in exactly the right stocks and invest in exactly the right time and you can fund projects on your own without alerting anyone. If you keep fantasies like that aside, then I think that in the end, even if this world ends up having less time, it was the right thing to do rather than just letting everybody sleepwalk into death and get there a little later. If you don’t mind me asking, what has being in the space in the last five years been like for you? Or I guess even beyond that. Watching the progress and the way in which people have raced ahead?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.