Tim Scarfe: evaluation

Transcript context

…the idea there is whatever goal it is that the AI has when it, you know, achieves this sort of basic level of of intelligence. Whatever that goal is, that goal will be served better by having more intelligence. And that intelligence in in Yadkowsky's worldview is something that you can increase by you know throwing more computing power and memory and whatnot at it. And the the idea there is no matter what your goal is, it could be something, you know, in the in the famous thought experiment from Bostrom, it could be something as silly as as, you know, creating more paper clips or it could be, you know, keeping people in, you know, engaged in chatbot conversations, or it could be, you know, making as much money as you can on the stock market. No matter what your goal is, the the thought with this with this notion of instrumental conversions is all of those goals will lead you towards seeking more power, more intelligence, more physical and computational resources in order to better achieve those goals and prevent barriers from being placed in the way of achieving those goals. I don't believe that's really true, but, not in the way that they would need it to be in order for this argument to go through. But that's that's the claim. Yeah. And it it seems to be an abstraction problem again because in in the the biological world, intelligence and agency are convolved in a very complicated way. Yep. Whereas in the in this sense, we're we're now abstracting them because there's also this orthogonality thesis as well, which is this intelligence of final goals are are, you know, disconnected from each other. But but, you know, more broadly on on on intelligence, it's almost like it's this magical abstract substrate that you can it it's on a single axis and you can you can just have more of it and more of it and but that's not really how intelligence works, is it? No. It's not. Not at all. Intelligence is quite a bit more complicated than that and is not, you know, a single valued thing. In fact, I'm not sure that we have a great definition of intelligence. And I also think that it's very clear that there are a great number of goals that, you know, don't work this way. I mean, maybe maybe it's just because it's, it's, you know, like I woke up in a cynical mood this morning, but I'm not convinced that the goal of being happy is 1 that is served well by being more intelligent or powerful.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.