Tim Scarfe: preference

Transcript context

…So the trick is that, Okay, so I've got this agent. And I know exactly what it does, Right? It takes into account information. Rolls out future internally, it rolls out a whole bunch of future consequences of various different actions or plans that it could take. It selects the best 1, and then it executes it. So all of those variables that occurred inside, from the outside perspective, it just looked like a function transformation. Unless I'm somehow going in and recording and somehow demonstrating the fact that the manner in which it is calculating its policy involved doing those rollouts, I wouldn't be able to show that it's actually doing those rollouts. I would just be able to conclude it has a really sophisticated policy. So can you conclude that something isn't is is so the question is how do you identify something is actually doing planning? And I think that's a really hard question as opposed to having an incredibly sophisticated policy. I think my intuition is if it feels to me that a function, a simple input output mapping can't be an agent. And in a way, is related to what we were talking about with grounding. It seems that when things are physically embedded in the world, then they're more likely to be agents. This functionalist idea that's just a bit of computer code running on a machine, it kind of feels like that can't be an agent. It does. So suppose I coded it up so it was doing all of that planning. It's like gets its inputs to some crazy, like massive Monte Carlo tree search, picks the best policy possible, and then executes it. Now you don't observe any of that. Right? Because you know what's going on, you could say, oh, well, it's it's clearly like executing, you know, this is it's doing planning and counterfactual reasoning. It's going on, like, look, there it is. Because you coded it, so you know it's doing it. But if you're looking at it from the outside, right, it you know, if you don't know what's happening inside, it's going you know, all you have access to is, oh, here's the action that it that it that it did given this long series of inputs. And so it's it's really hard to identify what, you know, something as an agent per se from the outside. You kinda have to know what's going on inside. This, by the way, is why I don't think that, like, you know, can you know, these sort of prediction based approaches to, AI are you know, you could sort of say, well, it it's not really doing anything even remotely agentic unless it's executing and doing planning counterfactual reasoning. So, like, your chess program is is like, oh, clearly, it's doing some planning and counterfactual reasoning because you know it's doing it. But but it but you could like write I could describe the exact same set of behaviors just with the policy function.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.