Guillaume Verdon: prediction

Transcript context

…But nevertheless, it’s true that humans are biased, grounded in our evolutionary biology, scared of everything that can kill us, but we can still imagine different trajectories that can kill us. We don’t know all the other ones that don’t necessarily, but it’s still I think, useful combined with some basic intuition grounded in human history, to reason about like what… Like looking at geopolitics, looking at basics of human nature, how can powerful technology hurt a lot of people? It just seems grounded in that, looking at nuclear weapons, you can start to estimate p(doom) maybe in a more philosophical sense, not a mathematical one. Philosophical meaning like is there a chance? Does human nature tend towards that or not? I think to me, one of the biggest existential risks would be the concentration of the power of AI in the hands of the very few, especially if it’s a mix between the companies that control the flow of information and the government. Because that could set things up for a sort of dystopian future where only a very few and an oligopoly in the government have AI and they could even convince the public that AI never existed. And that opens up sort of these scenarios for authoritarian centralized control, which to me is the darkest timeline. And the reality is that we have a data-driven prior of these things happening, right. When you give too much power, when you centralize power too much, humans do horrible things, right. And to me, that has a much higher likelihood in my Bayesian inference than Sci-Fi based priors, right, like, “My prior came from the Terminator movie.” And so when I talked to these AI doomers, I just ask them to trace a path through this Markov chain of events that would lead to our doom and to actually give me a good probability for each transition. And very often there’s a unphysical or highly unlikely transition in that chain, right. But of course, we’re wired to fear things and we’re wired to respond to danger, and we’re wired to deem the unknown to be dangerous, because that’s a good heuristic for survival, right. But there’s much more to lose out of fear. We have so much to lose, so much upside to lose by preemptively stopping the positive futures from happening out of fear. And so I think that we shouldn’t give into fear, fear is the mind killer, I think it’s also the civilization killer. We can still think about the various ways things go wrong, for example, the founding fathers of the United States thought about human nature and that’s why there’s a discussion about the freedoms that are necessary. They really deeply deliberated about that and I think the same could possibly be done for AGI. It is true that human history shows that we tend towards centralization, or at least when we achieve centralization, a lot of bad stuff happens. When there’s a dictator, a lot of dark, bad things happen. The question is, can AGI become that dictator? Can AGI when develop, become the centralizer, because of its power? Maybe because of the alignment of humans, perhaps, the same tendencies, the same Stalin like tendencies to centralize and manage centrally the allocation of resources? And you can even see that as a compelling argument on the surface level. “Well, AGI is so much smarter, so much more efficient, so much better at allocating resources, why don’t we outsource it to the AGI?” And then eventually whatever forces that corrupt the human mind with power could do the same for AGI. It’ll just say, “Well, humans are dispensable, we’ll get rid of them.” Do the Jonathan Swift, Modest Proposal from a few centuries ago, I think the 1700s, when he satirically suggested that, I think it’s in Ireland, that the children of poor people are fed as food to the rich people and that would be a good idea, because it decreases the amount of poor people and gives extra income to the poor people. So on several accounts decreases the amount of poor people, therefore more people become rich. Of course, it misses a fundamental piece here that’s hard to put into a mathematical equation of the basic value of human life. So all of that to say, are you concerned about AGI being the very centralizer of power that you just talked about?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.