Nathan Labenz: commitment

Transcript context

…ey're building out a lot of compute. So presumably they could throw a million human researcher equivalents at problems. And they also, by the way, you may have noted they run faster and they run 24 7. So the hope is that this will allow them to move much faster than they have moved and kind of pull away from the competition. I would say most people at that event thought that was very credible. There was not too much debate around like, will this level off? r than they have moved and kind of pull away from the competition. I would say most people at that event thought that was very credible. There was not too much debate around like, will this level off? Now, obviously there's some selection effect there, but you could just go to, the whole event was under Chatham House rule, so I will respect that and not attribute specific statements to specific people or organizations. But you could go to the recursive website to look at speakers, whose identities were shared obviously with their permission. And, they've definitely got some notable people from the frontier companies. So these were not people that are fringe or, who you would say likely don't represent, kind of mainline views at the companies. It really seemed that the expectation is, yes, this is going to work. It's going to have a major accelerating effect. We don't necessarily know if it's going to have a sort of simple accelerating effect. Like in a human organization, if you went from 1000 to a million researchers, you probably wouldn't get 1000 X output. So there may be some sort of, you know, coordination challenges or just kind of duplication challenges that we see in human organizations. Maybe that happens in the same way. That's one possibility where you still get acceleration, but it's not a like, you know, blinding kind of takeoff acceleration. Or, you know, I would say also understood to be a credible, realistic possibility was that it is even a more profound phase change than that. And things like pre-training just become dramatically more efficient and models, suddenly have all these new qualitative abilities that they didn't used to have, such as, continual learning that really works or what have you. And so everything could change, in a very dramatic way, potentially very quickly once these milestones are hit. In the room, people said, and it did, there was quite a distribution. I was pretty much right at the median when we were asked, how many copies of you would it take to do the work that you are currently doing with the benefit of AI? The median answer was basically two. In other words, people felt like they're getting two times as much work done thanks to AI. But that was also framed in an interesting way where it was like, but note that as of today, at least, if you were not there, your productivity would drop to close to 0. Not too many people felt that they had any system that would continue to work in any sort of meaningful way if they were entirely removed from the picture. So there's a significant productivity boost, but there's still this sort of necessity of at least some human salt into the recipe to get the whole thing working. And then a big part of the discussion too was like, how can we set that up in such a way or create some sort of self-correcting structure or some sort of governance mechanism that can keep that on the rails, broadly speaking? By far, the number one strategy seems to be monitoring. h a way or create some sort of self-correcting structure or some sort of governance mechanism that can keep that on the rails, broadly speaking? By far, the number one strategy seems to be monitoring. It's very, we're very, very, as a civilization, whether we know it or not, listening to people at the Frontier Labs who are about to, in their own minds, and I believe they're probably right, set off this relatively uncontrolled experiment of AI recursive self-improvement. The big thing that they are betting on is AIs monitoring other AIs. It's very like monitoring the chain of thought, watching out for bed stuff, maybe training some different models. One of the interesting ideas that I heard there that I had not heard before was that the model that you would want to have internally for AI research might have quite a different constitution from the one that you deploy publicly for kind of general purpose AI assistant use cases. And they seem to think that, in fact, you probably would want to have something even more focused on safety and more sort of restricted in some ways, but maybe also less inclined to refuse certain tasks, but basically a different behavioral profile, which I do think is interesting because if you're going to make this sort of chain of thought monitoring plan work, I do think you're probably going to need some meaningful diversity of the AIs. Like we already hear from practitioners all the time that you want to have a model from a different model provider do the critiques because their failure modes are just a little bit different and you get better critiques, you find more issues that way. So they are thinking that way a bit internally, but they're very, they're very focused on this phenomenon, making it happen, figuring out some ways, hopefully to kind of keep it on the rails. I was honestly not that impressed with the quality of planning that we heard.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.