Kenneth Stanley: belief

Transcript context

…Mhmm. Yeah. Yeah. That that that that's that's how this happened. I mean, the people people in general have intuitions about potential, not just about where we are right now, but that this might lead to something. And in fact, it's very complex because, like, the more people play with pick breeder, the more they get intuitions about pick breeder itself. So they start to understand what might lead to what what's promising, which is different than what's aesthetically pleasing in in the moment. And so that they're they're they're both enjoying the image and predicting, like, well, where might this go? And it's not like they're predicting they're gonna get a skull, but they're predicting symmetric things are really interesting and beautiful, and and and let's see what other kinds of symmetries come out of this, like they're thinking that. And so that helps to to get this kind of virtuous ordering, which causes like a a sequence of lock ins of different conventions of of increasing complexity, which then creates this, like, amazing representation underneath the hood. But I wanna I wanna point out though that, like, this view of the world is so radically different from the data driven view that we live in right now. You know, it's what's what's really fascinating to me about it is not data driven. Like, you know, we we think about, you know, when you say, like, eventually, if you experience enough of the world, like you say, of course, you might expect that your representation start to mirror just the way that the world is. And you get this kind of isomorphism maybe between, like, the the the the organization of your brain and the organization of outside reality. Well, that's a data driven view, like, in basically almost everything we think about is data driven. I mean, the bitter lesson is sort of like a data driven philosophy. But, like, this is, like, what's so interesting is this is totally contra bitter lesson. You know? Because like what you're saying here is like, we see almost nothing of the world, like, PickReader knows nothing of the world. There's no pre training at all. Start with some blobs. Over a few dozen iterations, I mean, dozen is crazy. Like, it's like peanuts. Like, we're used to millions, billions, like, we we don't do dozens. That's not what we do in our field. But here we have dozens of iterations, not enough to be exposed to almost anything. It actually somehow finds within this can newly constrained space of dimensions that has been discovered by humans, things like the difference between opening and closing a mouth or smiling and not smiling. And like those dimensions exist now in this space of the network, but not because of data. They're whole cloth de novo discoveries, which are not data driven. So there's no bitter lesson. It's just out of nothing. And there's even more crazy ones. Like, there's, like, like, the apple 1, which is in the paper. It's in the appendix. which are not data driven. So there's no bitter lesson. It's just out of nothing. And there's even more crazy ones. Like, there's, like, like, the apple 1, which is in the paper. It's in the appendix. This apple has this unbelievable weight, a single weight in the apple representation, which is a single continuum that if you move along that continuum, you swing the stem of the apple back and forth, like, from left to right. Maybe someday we'll put an animation over this so we could show like, I could give you the animation. But you can see the swinging stem is 1 dimension, and it's it's like three-dimensional. You know? It's not just like a 2 dimensional thing. Like, it moves like you would expect in a rotation in three-dimensional space. It has a shadow underneath it. It's like a green leaf. The underlying apple, which is a symmetric object, is not disturbed at all. It's totally independent. It's been decomposed. And then there's this 1 thing which is the stem swinging. And so, you know, what I'm saying is it's absolutely incredible. It's mind blowing that if you think about that as a world model, like, it's an actual true hypothesis about the world. This is the way that stems look when they swing. But this model has not been trained on anything in the world. It's never seen swinging stems, let alone apples at all. I can almost guarantee you that in the training trajectory itself, there was no swinging of the stem. After all, if the stem started swinging, that would mean it already had that ability. So it's just circular to argue that. This is just something that arose out of the fact that the that the representation is so elegant that it somehow has an internal hypo I think of it as a hypothesis about the world, which is correct. And you have to ask yourself, how many of our hypotheses are like that instead of data driven hypotheses? Because we do sometimes have these unbelievably elegant underlying representations of the world that often are unique. Unique to an individual, not necessarily universal across human beings. Like, everyone's representation is unique. And so this is a totally different way of thinking about representation and knowledge and how it comes to be.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.