Dan Hendrycks: prediction

Transcript context

…Yes. I mean, it's another 1 of those things where philosophically, I I agree with you. If such if such a recursively improving intelligence existed, I mean, God knows just to control it, we would lose control because we would have to use another recursively improving superintelligence to control the other 1, and then we would basically just be minnows in in the grand scheme of things. Yes. Destabilizing. If they control it, if if another state does this, you're in big trouble because if they control it, they can weaponize against you. And if they don't control it, which I think would be the more likely outcome because they'd be doing it under extreme time pressures, cutting a lot of corners. They wouldn't like they'd be operating with very high risk tolerance. If they're doing it very slowly, then they would probably be needing to coordinate with others or else they're not seeing an edge in doing it. So I think that it'd be operating with an extremely high risk tolerance if they're doing a fully automated R and D loop. So yeah, yeah. So I think loss control risks from recursion are very high and shouldn't be pursued. And I think it's very interesting that AI companies talk about this sort of stuff openly. I think that there's something wrong with, I think, the norms for that. Because a lot of them also acknowledge that, oh yeah, we don't really have a plan for how to control that. And we don't really think we will. But that's that's the plan. I I I I think something's broken. But yeah. Yeah. I mean, another thing I'm interested in is is open ended systems in general. So e evolution is this fascinating open ended system, which is constantly creating new niches, new problems, and solutions in in tandem. But it seems to have converged. I mean, human intelligence, I think, has actually peaked and and gone down a little bit. A corporation is a collective intelligence, and that seems to have reached a limit. Agenetic forms of AI seem to peak. I mean, I'm I'm I'm really interested in open ended algorithms like POET, you know, the the pairwise open ended Trailblazer or even Sakana AI. They did this thing on ARC last week, and they just basically created this Monte Carlo tree search type thing where they found, you know, switching expert trajectories of different foundation models, generating code, and testing the code on ARC challenges. And the the the common theme you see is convergence. So in in the Sokana paper, after 250 calls, it converged and it failed to expand and to improve its results. And would would you agree? I mean, like, we we can agree that there's some margin. Right? You can improve to some margin, and we don't know what that margin would be if we had, like, agentic superintelligence. But do do do you think that margin would converge quite quickly, or do we we just don't know?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.