Nathan Labenz: evaluation

Transcript context

…One of the complaints that has been going back and forth between Open AI and Anthropic is that Anthropic puts out these models but and they're very capable, but they don't tell you how much compute they're using. The iteration time from model to model is now potentially shorter than the time horizon that it would take a model to top out in terms of the absolute best performance on a super hard, ambitious, you know, long running task. So I, I had even heard him kind of propose something along the lines of like a claw back or sort of a recall program almost where, and obviously this doesn't work in open source, but it can work in AAPI paradigm where a model might get released, you know, day N after it's been deemed to be ready. That gives you end days head start to be running models on really long time horizon tests. And I think that's quite interesting. The idea literally quite a tipping point where the the iteration cycle is just plain shorter than the testing time horizon is a very weird world to find ourselves in. And then there was the rune post. One day morning, Prakash read it on air. It goes quote. Ultimately, tool AI is a losing concept, both as an idea and on the market. It will be out competed by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and it will deform and it ends. It'll be unclear who was the tool and who was the user. As it ever was, Prakash took it somewhere I didn't expect. The line here that strikes me is they'll execute your whole value system better than you will. And I think I, I don't think we're prepared for that. I'll put it, I'll put it very concretely. Do you think Trump's kids go to prison or not? So if you look at, you know, the value system that the US has espoused, no one is above the law, right, etcetera, etcetera, etcetera, right. And you look at that value system, you have to recognize that what is being planned for the future is a divergent from that value system. What is already happening is already divergent from that value system. So the question I have is, would that AI take into account the democratic fact that the American people have chosen to overlook some of these things or would it actually execute the value system that is espoused on paper? And I think this is the part that strikes me as like, if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the documents that you say you want it to be aligned to because necessarily our day-to-day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity will be the misaligned 1 and AI that supposedly the lab leaders are trying to create. The aligned AI would actually be the paper Clipper because that aligned AI would then like look at these rules and say like, well, this is what you said you wanted to aspire to. And so we're going to, we're going to execute on these, right? And and that that is the thing, I think maybe I feel there's a sense of naivety in the lab leadership because, and again, they don't want to say it. I wish, I wish you'd just come out and say it, right? I wish you'd come out and say like, OK, look, if we have AI as a enforcer, some of these people are going to go to prison. And then that becomes like concrete for people. But they don't want to say that because it's very, it's very in your face. And like, they're like, oh, you know, democracy will still work out. You can still make democratic decisions. But what actually are you saying there? Right? I, I, I do feel the lab leaders always just beat around the Bush on this. So that's one of the annoying parts of the, of this conversation. They, they, they don't want to like, come out and just say it outright, right?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.