Scott Alexander: belief

Transcript context

…What about the skepticism that, look, what you’re suggesting with this hyper efficient hive mind of AI researchers, no human bureaucracy has just out of the gate worked super efficiently, especially one where they don’t have experience working together. They haven’t been trained to work together, at least yet. And there hasn’t been this outer loop RL on like, “we ran a thousand concurrent experiments of different AI bureaucracies doing AI research and this is the one that actually worked best”. And the analogy I’d use maybe is to humans in the Savannah 200,000 years ago. We know they have a bunch of advantages over the other animals already at this point, but the things that make us dominant today, joint stock corporations, state capacities like this fossil fueled civilization we have that took so much cultural evolution to figure out. You couldn’t just have figured it out in the savannahs like, “oh, if we had built these incentive systems and we issued dividends, then we could really collaborate here” or something. Why not think that it will take a similar process of huge population growth, huge social experimentation, and upgrading of the technological base of the AI society before they can organize this hypermind collective, which will enable them to do what you imagine an intelligence explosion looks like? Yeah, you’re comparing it kind of to two different things. One of them is literal genetic evolution in the African savannah, and the other is the cultural evolution that we’ve gone through since then. And I think there will be AI equivalents to both. So the literal genetic evolution is that our minds adapted to be more amenable to cooperation during that time. So I think the companies will be very literally training the AIs to be more cooperative. I think there’s more opportunity for pliability there. Because humans were, of course, evolving under this genetic imperative that we want to pass on our own genetic information, not somebody else’s genetic information. You have things like kin selection that are kind of exceptions to that, but overall it’s the rule. In animals that don’t have that, like eusocial insects, then you very quickly get, just through genetic evolution, without cultural evolution, extreme cooperation. And with eusocial insects, what’s going on is that they all have the same genetic code, they all have the same goals. And so the training process of evolution kind of yokes them to each other in these extremely powerful bureaucracies. We do think that the AI will be closer to the eusocial insects in the sense that they all have the same goals, especially if these aren’t indexical goals, they’re goals like “have the research program succeed”. So that’s going to be changing the weights of each individual AI, I mean, before they’re individuated, but it’s going to be changing the weights of the AI class overall to be more amenable to cooperation. And then, yes, you do have cultural evolution. Like you said, this takes hundreds of thousands of individuals. We do expect there will be these hundreds of thousands of individuals. It takes decades and decades. Again, we expect this research multiplier such that decades of progress happen within this one year, 2027 or 2028. So I think between the two of these, it is possible. Maybe this is also where the serial speed actually does matter a lot. Because if they’re running at 50x human speed, then that means you can have a year of subjective time happen in a week of real time. And so these sorts of large scale cooperative dynamics of your moral maze, you have an institution, but then it becomes like a moral maze and it sort of collapses under its own weight and stuff like that. There actually is time for them to play that out multiple times and then train on it, tinker with the structure and like add it to the training process over the course of 2027.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.