Tim Scarfe: evaluation

Transcript context

…Yeah. Yeah, mean, think the notion of agency is an interesting 1 where I really am amenable. I mean, is sort of a I don't know if I would call it a schism in the field, but there is a debate where between sort of the reinforcement learning world and the active inference worlds where how much of intelligence can be conceived as optimizing a reward function. And sort of the hardcore reinforcement learning world is like everything is just a reward. And then the active inference world would argue that not all behavior is driven just by optimizing a single reward function. There is some uncertainty minimization. There is trying to satisfy your own model of yourself, fulfill your own predictions, these sort of things that seem very well aligned to behavior we see. But, you know, it's unclear which of these is right. It's probably some balance of the 2. But to me, agency, people would conceive of agency differently in these 2 worlds. Right? So I think some people in the RL world would say agency is just you give something a reward function, and then it just learns over time trying to optimize that reward. In the more active inference world, which I do think has legs and I'm obviously amenable to, the idea of agency is a little bit more. It's building a model of yourself and trying to infer what your goals are based on observing yourself and then trying to make predictions to fulfill those end goals. In other words, it's constructed, goals are constructed. And this is sort of 1 of my favorite Friston papers is predictions not commands. I don't if you've read that paper, but I think it's a brilliant paper about how you could reconceive motor cortex not as sending motor commands to your body, but actually as building a model of yourself and predicting what will happen. And the way the spinal cord is wired is it just fulfills those predictions. And I think that's a really interesting sort of reframe of how you could get agency and really interesting smart behavior in the absence of just a strict reward function, right? So how that would learn is it's trying to model the behaviors it observes, then it's trying to sort of predict those and fulfill them. So but yeah, I think agency is a really interesting concept because it sort of manifests itself in these different paradigms in different ways. Yeah. I I find it fascinating. I mean, because the way I read it in active inference literature, it's a very principled definition of an agent. And there's still a bit of a gap because I think Friston would argue, in the natural world, because of the the laws of physics and particles and and whatnot you get the emergence of things and things become agents when they have a certain you know depth of planning shall we say. Yep. But yeah it's it's really, really interesting. And I guess he would argue that you get all of these, phenomena that give rise to agency like biotic self organization and and and so on. But, yeah, maybe we should do we'll we'll we'll slowly go go in that direction. So you you give the example of, mice doing planning. Can you can you sketch that out? Yeah. So this is another real, like, area of neuroscience research that I just absolutely love. So it was the case, I think it was the forties or fifties, I forget the exact decade, where Tolman observed that worms when they, mice, when they would reach choice points in mazes where he was training them to navigate around mazes, would pause and then they would sniff back and forth and then they would choose an action. And so he hypothesized this idea that they must be engaging in vicarious trial and error. They must be imagining possible outcomes before deciding. And of course, this was hugely controversial because there's no evidence he had no evidence that they were in fact imagining anything, and of course, people like to, in the absence of evidence, assume animals are as dumb as possible. Only when there's irrefutable evidence will we imbue them with any intellectual capacities, which I think is an interesting human bias, but that's fine. And then David Reddish, who is also a close friend and mentor of mine, he did some amazing research with 1 of his PhD students where they were recording hippocampal place cells. So it's a very quick background for viewers. You can go into the hippocampus of really any mammal, but this is best well studied in rats. And part of the hippocampus, a region called CA1, has these things called place cells and so if you look at, if you record it as a mouse is moving around a maze, what you find is this incredible thing where there are neurons that activate only in specific locations in that 2 dimensional plane And it's not based on how they got there. It is egocentric it is independent of their egocentric path. It is allocentric, meaning in the plane of external space. So they can come back to the same place from any route and that same place cell will activate. And as an aside from the evolutionary story, we find similar types of cells, it's not exactly as accurate, but we find similar types of cells in fish, in the homologous region of the hippocampus of their cortex where they have place like cells, it's not as accurate but cells that activate in certain locations in a maze. What he found is when mice engage in this act of vicarious trial and error, meaning when they pause and look back and forth, the place cells in the hippocampus cease to only activate in the location they are, it actually starts activating down the paths of each route it might take. In other words, you can literally watch rats imagining the future. I think 1 of the most incredible neuroscience findings. And so he then took this and then did a bunch of other experiments which I think reveal even further the power of sort of imagination in rats. 1 of my favorites is his counterfactual learning studies where he puts rats in this thing called restaurant row where it's a sort of, square like yeah. Exactly. It's just like a square like maze.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.