Scott Alexander: recommendation

Transcript context

…So five years, if they’re going at 50x serial speed, then five years is what? Like 250 years of serial time for the AIs, which to me feels like more than enough to really sort out this sort of stuff. You’ll have time for sort of like empires to rise and fall, so to speak, and all of that to be added to the training data and yeah. But I could see it taking longer than we depict. Maybe instead of six months, it’ll be like 18 months, you know, but also maybe it could be two months. So when I think of the ways that they train AIs, I think in our scenario at this point there are two primary ways that they’re doing it. One of them is just continuing the next token prediction work. So these AIs will have access to all human knowledge, they will have read management books in some sense, they’re not starting blind. There is going to be something like: predict how Bill Gates would complete this next character or something like that. And then there's reinforcement learning in virtual environments. So get a team of AIs to play some multiplayer game. I don’t think you would use one of the human ones because you would want something that was better suited for this task. But just running them through these environments again and again, training on the successes, training against the failures, kind of combining those two kinds of things. To me it does not seem like the same kind of problem as inventing all human institutions from the Paleolithic onward. It just seems like applying those two things. The other notable thing about your model is, you got this superhuman thing at the end of it and then it seems to just go through the tech tree of mirror life and nanobots and whatever crazy stuff. And maybe that part I’m also really skeptical of. If you look at the history of invention, it just seems like people are just trying different random stuff, often even before the theories about how that industry works or how the relevant machinery works is developed; like the steam engine was developed before the theory of thermodynamics, the Wright brothers seemed like they were just experimenting with airplanes, and is often influenced by breakthroughs in totally different fields. Which is why you have this pattern of parallel innovation, because the background level of tech is at a point at which you can do this experiment. Machine learning itself is a place where this happened, right? Where people had these ideas about how to do deep learning or something. But it just took a totally unrelated industry of gaming to make the relevant progress, to get the whole, basically the economy as a whole advanced enough that deep learning, Geoffrey Hinton’s ideas could work. So I know we’re accelerating way into the future here, but I want to get to this crux.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.