Tim Scarfe: evaluation

Transcript context

…Well, mean, clearly clearly, as I, argue in the book, I don't think that the brain is just 1 big transformer. So I would agree with you. In in the human brain, unless you think there's something that's nondeterministic and sort of magical happening, you know, I think you would still say that there is either, you know, base firing rates of neurons, that and then there's sensory input that flows up and then goes through the brain until eventually there's sort of it's affecting muscles until you're responding. So there is, you know, there is sort of a deterministic flow happening. It might not be as feed forward as what's happening in transformer which is definitely the case, but they might both be sort of, you know, deterministic in in a similar fashion. I think a lot of people I've had this exact sort of argument with a lot of people and 1 counterargument that people have towards this idea, I don't know if I fully agree with it, but it's interesting, is that attention heads really are doing something more magical than we give them credit for, which is they are kind of dynamically rerouting and effectively resetting the network based on the context that the prompt is getting. And so although technically it's just a series of, you know, matrix multiplications, etcetera, if in principle what's happening is these attention heads are doing something really clever where they're looking at the context of a prompt and then effectively dynamically reweighting the network to decide what it cares about and what it doesn't. So there are people that think, you know, there is something really interesting happening in the transformer that might be analogous to certain things that are happening in the brain. But, yeah, clearly, you know, these these feed forward networks are are not capturing everything that's going on in the brain. Yeah. It's really interesting what you said because the way I read that is things like chat GBT and language models, they are entropy smuggling or agency smuggling. So what what that means is they kind of just do what you tell them to do and all of the kind of, the agency, so my directedness comes from me. So I give it a prompt, it does the thing that I wanted it to do And then the kind of the the mapping that you were talking about, I interpret that a bit like a database query. So, you know, depending on the prompt you give it, it'll activate a certain part of the representation space, and it will give you a certain result back. But the the the brain has this thing where all of the neurons have their own directedness. And and the weird thing is at the cosmic scale, agency is a site it seems to emerge. So even transformer models that were acting autonomously could presumably in large enough scale give rise to something that we think of as directedness or goals or purpose or or or whatever. But it's almost like, in the natural world, because there are so many levels, scales and scales of independent autonomous things just kind of mingling with each other independently, and then, like, downstream mixing their information together and rinsing and repeating over many, many different scales. That seems to be the thing that gives rise to all of these amazing things like agency and creativity and and etcetera. Yeah. Yeah, mean, think the notion of agency is an interesting 1 where I really am amenable. I mean, is sort of a I don't know if I would call it a schism in the field, but there is a debate where between sort of the reinforcement learning world and the active inference worlds where how much of intelligence can be conceived as optimizing a reward function. And sort of the hardcore reinforcement learning world is like everything is just a reward. And then the active inference world would argue that not all behavior is driven just by optimizing a single reward function. There is some uncertainty minimization. There is trying to satisfy your own model of yourself, fulfill your own predictions, these sort of things that seem very well aligned to behavior we see. But, you know, it's unclear which of these is right. It's probably some balance of the 2. But to me, agency, people would conceive of agency differently in these 2 worlds. Right? So I think some people in the RL world would say agency is just you give something a reward function, and then it just learns over time trying to optimize that reward. In the more active inference world, which I do think has legs and I'm obviously amenable to, the idea of agency is a little bit more. It's building a model of yourself and trying to infer what your goals are based on observing yourself and then trying to make predictions to fulfill those end goals. In other words, it's constructed, goals are constructed. And this is sort of 1 of my favorite Friston papers is predictions not commands. I don't if you've read that paper, but I think it's a brilliant paper about how you could reconceive motor cortex not as sending motor commands to your body, but actually as building a model of yourself and predicting what will happen. And the way the spinal cord is wired is it just fulfills those predictions. And I think that's a really interesting sort of reframe of how you could get agency and really interesting smart behavior in the absence of just a strict reward function, right? So how that would learn is it's trying to model the behaviors it observes, then it's trying to sort of predict those and fulfill them. So but yeah, I think agency is a really interesting concept because it sort of manifests itself in these different paradigms in different ways.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.