Tim Scarfe: evaluation

Transcript context

…It does. So suppose I coded it up so it was doing all of that planning. It's like gets its inputs to some crazy, like massive Monte Carlo tree search, picks the best policy possible, and then executes it. Now you don't observe any of that. Right? Because you know what's going on, you could say, oh, well, it's it's clearly like executing, you know, this is it's doing planning and counterfactual reasoning. It's going on, like, look, there it is. Because you coded it, so you know it's doing it. But if you're looking at it from the outside, right, it you know, if you don't know what's happening inside, it's going you know, all you have access to is, oh, here's the action that it that it that it did given this long series of inputs. And so it's it's really hard to identify what, you know, something as an agent per se from the outside. You kinda have to know what's going on inside. This, by the way, is why I don't think that, like, you know, can you know, these sort of prediction based approaches to, AI are you know, you could sort of say, well, it it's not really doing anything even remotely agentic unless it's executing and doing planning counterfactual reasoning. So, like, your chess program is is like, oh, clearly, it's doing some planning and counterfactual reasoning because you know it's doing it. But but it but you could like write I could describe the exact same set of behaviors just with the policy function. I think the counterfactual thing is an important feature here because we could take something which was conscious or something which had agency, and we could just take a trace of the actual path which was found. And now we've just got this is reductio ad absurdum. But now we've just got a computational trace. And that thing clearly has now lost whatever agency or consciousness it had. So there's something about considering all of the possibilities. Yeah. Yeah. I think so in my mind, that is the fundamental feature of an agent. Like if you can show that it's engaged in planning counterfactual reasoning, then it's definitely an agent. My argument is just simply that that's hard to do unless you crack it open and see what's going on inside. Now you could take a a pragmatic view and say, well, if the simplest computational model of the behavior, model it as if it was doing planning and counterfactual reasoning, then you can draw an implicit conclusion that, oh, yes, well, I may as well say it's an agent. And that's kind of the approach that I've taken. So like 1 of the things that comes out of the physics discovery algorithm is that you apply it to agents and what do you get? Well, you get a model. Now bear in mind, I called them all objects before, and I didn't change anything to make it special to an actual agent. Right? But what I do have the ability to do because of the model is I can look at the internal states associated with that object that I want to call an agent, and look at how sophisticated it is, right? And that degree of sophistication is what allows me to say, oh, well, I'm going to go ahead and say that and I like the whole idea. It's a great idea. Let's have a metric, right? And I'm sure it would be something that would effectively be transfer entropy or something like that. But we have this metric on, well, how sophisticated were the internal states that were necessary in order to generate this output? And if it's above some threshold, we'll call it an agent. I don't like thresholds. But we just sort of say a degree of agency, a degree of sophistication.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.