Tim Scarfe: uncertainty

Transcript context

…Yes. Destabilizing. If they control it, if if another state does this, you're in big trouble because if they control it, they can weaponize against you. And if they don't control it, which I think would be the more likely outcome because they'd be doing it under extreme time pressures, cutting a lot of corners. They wouldn't like they'd be operating with very high risk tolerance. If they're doing it very slowly, then they would probably be needing to coordinate with others or else they're not seeing an edge in doing it. So I think that it'd be operating with an extremely high risk tolerance if they're doing a fully automated R and D loop. So yeah, yeah. So I think loss control risks from recursion are very high and shouldn't be pursued. And I think it's very interesting that AI companies talk about this sort of stuff openly. I think that there's something wrong with, I think, the norms for that. Because a lot of them also acknowledge that, oh yeah, we don't really have a plan for how to control that. And we don't really think we will. But that's that's the plan. I I I I think something's broken. But yeah. Yeah. I mean, another thing I'm interested in is is open ended systems in general. So e evolution is this fascinating open ended system, which is constantly creating new niches, new problems, and solutions in in tandem. But it seems to have converged. I mean, human intelligence, I think, has actually peaked and and gone down a little bit. A corporation is a collective intelligence, and that seems to have reached a limit. Agenetic forms of AI seem to peak. I mean, I'm I'm I'm really interested in open ended algorithms like POET, you know, the the pairwise open ended Trailblazer or even Sakana AI. They did this thing on ARC last week, and they just basically created this Monte Carlo tree search type thing where they found, you know, switching expert trajectories of different foundation models, generating code, and testing the code on ARC challenges. And the the the common theme you see is convergence. So in in the Sokana paper, after 250 calls, it converged and it failed to expand and to improve its results. And would would you agree? I mean, like, we we can agree that there's some margin. Right? You can improve to some margin, and we don't know what that margin would be if we had, like, agentic superintelligence. But do do do you think that margin would converge quite quickly, or do we we just don't know? Yeah. I don't know if it would saturate necessarily. I mean, certainly, if they're at some capability level, this might affect the rate of improvement substantially. But and obviously, there has to be a limit because of physics somewhere. But I would imagine there's a lot of room for improving its sort of ability to handle things of more and more Kolmogorov complexity, for instance. So like some of those sequences or in those Raven's Rasimetry, you can think of just as this arc type of things. Some have lower Kolmogorov complexity. Some have higher Kolmogorov complexity. It's just some notion from theoretical computer science. And I don't think humans are anywhere near the sort of computational limit for that at all. And I would guess that there'd be that number could keep going up. So I think fluid intelligence could get like, think basically the IQ of the AIs, wouldn't be all the intelligence, will be separate from their, you know, their their long term memory, from their visual processing ability, from their reaction time, etcetera. But but their but I think their IQ could get really high, and they could solve, like, extremely difficult mathematics problems and solve have really good intuitions for for puzzles and not take much compute for it. And I think that number could keep going up quite a bit. So, like, we're quite limited. Like, we've got big brains, but, like, you know, it's still like, there's not much hardware here. And you you can imagine a bigger brain that would have, I think, pretty substantial capabilities even for the intelligence of human across generations. Like, there's like Flynn effect and things like that. And that's that's that's sort of has greatly increased. So I we're we're our, you know, our brain size is quite limited by what can give what what is is can go through the the birth canal. And I I expect that AIs wouldn't have that limit at all. They could they could, at the very least, keep getting faster and faster. You still have Moore's Law. And you have GPU improvement rates, are, like, 2x every 3 years. And you also have scalability. You also have better ability to transfer state than it is with humans. Because a lot of this digital computation is is more precise than analog. So I think they have a variety of I think they have a you can you can better transfer things across generations as well. There's this Nobel Prize winners are not able to you know, their their sort of descendants kinda go downhill, generally. So I think it's I think that they have some pretty substantial advantages that could that could really compound and accumulate the number of if if you're saying that it's mostly if it's mostly externally computed, well, can you know, humans, how many how many connections can people do? They can do, like, I don't know, Dunbar's number, like, maybe a 130 or so social connections, meaningful social connections. They could do thousands, tens of thousands, millions.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.