Scott Alexander: belief

Transcript context

…I agree. We’re on a totally different topic here of, do you get a Dyson sphere? There’s one world where it’s crazy but it’s still boring, in the sense that the economy is growing much faster, but it would be like what the Industrial Revolution would look like to somebody in the year 1000. And that one is one where you’re still trying different things, there’s failure and success and experimentation. And then there’s another where the thing has happened and now you send the probe out and then you look out at the night sky 6 months later and you see something occluding the sun. You see what I’m saying? Yeah. So like we said before, I think there’s a big difference between discontinuous and very fast. I think if we do get the world with the Dyson sphere in five years, in retrospect, it will look like everything was continuous and everyone just tried things. Trying things can be anything from trial and error without even understanding the scientific method, without understanding writing, maybe without even having language and having to be the chimpanzees who are watching the other chimpanzees use the stick to get ants, and then in some kind of non-linguistic way this spreads, versus like the people at the top aerospace companies who are running a lot of simulations to find the exact right design, and then once they have that, they test it according to a very well designed testing process. So I think if we get the ASI and it does end up with the Dyson sphere in five years- and by the way, I think there’s only like 20% chance things go as fast as our scenario says. It’s Daniel’s estimate, it’s not my median estimate, it’s an estimate I think is extremely plausible that we should be prepared for. I’m defending it here against a hypothetical skeptic who says “absolutely not, no way.” But it’s not necessarily my mainline prediction. But I think if we do see this in five years, it will look like the AIs were able to simulate more things than humans in a gradually increasing way. So that if humans are now at 50% simulation, 50% testing, the AIs quickly got it up to 90% simulation, 10% testing, they were able to manufacture things much more quickly than humans so that they could go through their top 50 designs in the first two years. And then after all of the simulation and all of this testing, then they eventually got it right for the same reasons humans do, but much, much faster. In your story, you have basically two different scenarios after some point. So, yeah, what is a sort of crucial turning point and what happens in these two scenarios?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.