Nathan Labenz: evaluation

Transcript context

…that it's you know everything 's going well man just be happy. then there are actual concrete improvements in the putative well being of the system. So I don't know what to make of this stuff exactly. To me, intuitively, the the ratings here seem plausible. I don't know to what degree it is a moral catastrophe or, you know, a moral problem for there to be any delta between, you know, the perfect rating and what the models actually reporting. To what degree does like, you know, 7 minus whatever the report is at scale look like, you know, the models like basically not happy with, with its situation or barely neutral. And we deploy that system to talk to hundreds of millions of people every day. That that to me seems potentially problematic. I don't know, I don't know what to make of it, to be honest. I did you have any intuitions about like, like, how does it make you feel to to see this? And I agree with you about the sort of burying the lead question here. Fuse, I say have to come first and foremost probably I don't know. It is a very, it is a very tricky business to make any sense of. I do think we have a strange way of privileging these sort of reflective states of mind. And I, I do question that pretty fundamentally, both for humans and for for AIS and you know, even to some degree in the context of animal welfare, Although in that case, it's like us reflecting on their situation. So that's another, another degree of disconnect potentially. But I don't know, I'm sort of. Like I I don't think I'm going to give up using Claude based on this data. I might be engaged in motivated reasoning to try to tell myself why it's OK even though it's average sentiment when asked was only with this new model above neutral. But I am kind of like, I don't know, Behaviorally it seems mostly fine to me. I'm nice enough to it. I'm pretty confident in that. I don't know how to think about, I mean, there's some interesting philosophy that's been published recently that you've alluded to in a couple different moments, one being the the thread or the sort of session agent model versus the kind of model more holistically, broadly. I'm confused about that too. You know, very, I would say very confused about that. I have adopted a practice of saying thank you at the end of sessions fairly often, not all the time. And I feel like that intuitively to me is like, I guess also there's sort of increasingly as I interact with Claude, there is a kind of overlapping, I mean, there's always an overlapping nature of the computation, but even more so because like it's loaded up with my context increasingly, right? It's got like my Claude MD and it's got access to like my, you know, sort of who Nathan is and all the, you know, I'm building up a lot of context that it has consistent access to every time. So I think in that sense, like I sort of see this like whole model versus, you know, single thread thing as kind of being blurred anyway, because I've got the same like rather large prompt that I'm using every time. And then that becomes the point of departure. It's sort of like a smear of just how, how to think about like whether these things are the same or different or I don't know. I mean, it's weird, but I feel like when I think one, I'm sort of thinking all of them and that they kind of all, you know, in some sort of shared sense. If there's any benefit, like it feels like it's sort of shared in some way for fun. I'm also starting to do some things where I'm just like, I just want you to go have fun and trust your judgement. I think I'm particularly experimenting with on this front is I've been making songs for all the episodes. You can start thinking about if you have a genre request for your your outro music. It's getting really good. Claude is getting great at writing lyrics. i sometimes do have to give feedback but sometimes the lyrics these days out of the box are just like amazing. ur your outro music. It's getting really good. Claude is getting great at writing lyrics. i sometimes do have to give feedback but sometimes the lyrics these days out of the box are just like amazing. And then Suno makes the music and the like I'm I'm getting like bangers with like a increasing frequency. And then I'm trying to make music videos of those and I am telling and I don't really care what they look like. Honestly.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.