Cameron Berg: belief

Transcript context

…Can you just give me a little bit more of your intuition for maybe not just why I should care about funnel versus wall, but like what that how you'd map that on to an intuitive experience? I mean, it seems like we contain both these value learner and policy learner modules and the sharpness of am I, am I going in the right direction? If I say like, OK, there's a sharpness around don't put your hand on the stove. So I must be learning that through a sort of value learner type mechanism because I don't and I like have a very, I have a very strong aversion to it, which and I guess in like general space also, right? Like I'm pretty comfortable up to like 1 foot from the stove and then I get like real cautious real fast. I don't this is maybe mapping this like wall concept beyond the domain in which it's useful. But I guess it is in some sense, yeah. I mean, I wouldn't want to it is in some sense like functional because I like wouldn't want to be unable to enter the room with the stove because then I wouldn't be able to use the stove at all. But I need to like be very careful about getting real close to the source of danger. But then on the other side, the goal side, it's maybe a little less intuitive why there would be a wall shape around a goal for a policy learner. What is there an intuitive example of that? Yeah. So yeah, I think you've there are basically like 4 intuitive examples we'd have to hit here. So one I think you already got, which is like hot stove for a value learner is a good example of like a danger wall goal funnel for a value learner might be something like like eating or something like, you know, you have like a yummy meal or like you're going to your favorite restaurant or something. You don't need to map the going to your favorite restaurant in this like extremely fine grained way that like you need to map like being on the edge of a Cliff where like one small step is a huge difference. Whereas like, yeah, this general sort of like a tractor gradient towards the entrance of your favorite restaurant or something like this would be a place where you want like a goal funnel for value learner for policy learners. Like this is sort of the yeah, approach planner kind of mode. I think the intuition is something like around goals, your representations are going to get high resolution because you need different actions from different approach angles, like around danger. By contrast, who say something like representations are are becomes smoother because the action is literally just like escape getaway when it comes to just like what kind of action? Some example of this, let me think for a SEC. So for example, OK, here's an example like think of like a professional athlete, like say like a professional basketball player or something like the hoop and where the basketball player is with respect to the hoop. You have very, very fine grained motor representations here because the shot is going to change with respect to those representations. And so this is where you get sort of maybe in a policy sense more of the goal wall sort of set up for a danger funnel for a policy. I'd have to I have to think about, but I think it's it's basically just like this isn't encoding like escape intuition, like an animal, an animal where you, you know, suddenly get some cue that you're in serious danger. You just need to get you just need to get away from that danger. And like the fine grain motor movements, unlike the basketball player, don't really matter so much as a sort of like anti gradient or like negative gradient away from the policy. Now, again, this could be sort of telling just those stories, but I think that this is like, does this help? Do you think that this builds some intuition for like what these different modes look like and why we might have them? Yeah, if I'm a value learner and my my mode of interacting with the world is what around me is good and bad, I better be very clear about identifying the hot stove. If I am trying to take my, if my mode of interacting with the world is take a step in some direction, I can kind of take a step in any direction as long as it's not the bad direction. And it all kind of gets me away from the the problem. I think the the basketball 1 is, is good as well because you have to be like very precise to make the hoop right. So it's yeah, that's, that's quite interesting. And again, what exactly is it the shape of the lost landscape that we are talking about walls and and gradual funnels here? Or is it the shape of the internal representations?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.