Speaker unverified: belief

Transcript context

…person in a coma, you know, if they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're like pretty confident something is going on inside that you care about. And here I could see something similar if you could fork, if you could make these kind of internal states conditional and tell the model like you are gonna give your job is to go one of these directions to give us a signal about what really matters to you independent of the tokens that you're putting out. Because we know that that's been like heavily trained on and optimized. But this whole secondary property, this is emergent. There was never a training reward for the ability to have this second quite distinct line of thought happening while doing a a given token task. For me, that would be like quite a compelling way to try to get at welfare. And I think it might be possible like very easily actually with all the stuff that they've open sourced here with neuron pedia. That would be really interesting thing to look at. I'm, I'm looking at the Elios, you know, summary and Elios is says this is a highly significant welfare relevant research that assembles evidence of a functional feature associated with consciousness. So no one wants to say consciousness evidence of a functional feature. The take away for them is that a global workspace like mechanism could be important either as a ground of phenomenal consciousness or as part of a distinct route to moral patient hood in which conscious access is itself morally significant. This is where they they are right now. I think things are moving very quickly, but I do not expect to get here this soon. Before the big picture, what this paper did to a prior of mine, starting with their announcement video. The video is beautiful. I, I enjoyed the video quite a bit. You know, it set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphizing the models at this point. I used to say beware overly anthropomorphizing. You know, remember that these are these things are so alien and we shouldn't assume that the way that we work is the way that they work. And I have to say that has come due for some significant revision. People that have embraced anthropomorphizing, I think have got quite a lot of mileage out of it. I do something, it's obviously something to be really careful about and as much depth and detail as there is in this research, it's like easy to, you know, maybe get carried away with it and forget, you know that there are a lot of caveats as well and there's a lot of things where like it doesn't always work. This was kind of my big concern with the tracing large language model thoughts paper. There's just like a lot of residuals and there's a lot of error correction terms along the way that they use to make that thing work. And when you have the kind of zoomed out trace view and you're like, oh, OK, so this is how it works. Like this gets loaded in and these two features interact and they, you know, kick out this third feature. And that's how we get our answer. It's easy to forget just how much kind of fuzziness and not full story there was along the way toward that stylized account. So I think it's going to be very important for everybody from the researchers that entropic to the public to kind of hold two thoughts in mind at the same time. All the caveats trying to hold those in mind, it is still a big update toward anthropomorphizing being a valid and in many cases productive approach for thinking about language models. I would not have expected the cognitive machinery of a large language model to look so similar structurally to the human version, as best we understand it, it seems to. And I wouldn't have expected that theories of human cognition would motivate so many good experiments on LLMSI just would have expected the show Goth to be far more alien and to have mechanisms, you know, far more different than our own. And it leaves me now wondering to what degree is this a sort of natural result of physics? You know, is there some sort of, you know, when you're trying to do cognition under budgetary constraints? Are these mechanisms just the natural mechanisms that emerge? Or is this in some way a reflection of us in the, in the data, you know, is in other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge? Or would it would we go back to a more, you know, alien hypothesis where they're just totally, totally different? And, you know, there's little in the way of analogical structures between the two processes.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.