Carl Shulman: preference

Transcript context

…Okay, final question. How do you think about info hazards when talking about your work? Obviously if there's a risk you want to warn people about it but you don't want to give careless or potentially homicidal people ideas. When Eliezer was on the podcast talking about the people who've been developing AI being inspired by his ideas. He called them idiot disaster monkeys who want to be the ones to pluck the deadly fruit. I'm sure the work you're doing involves many info hazards. How do you think about when and where to spread them? I think they're real concerns of that type. I think it's true that AI progress has probably been accelerated by efforts like Bostrom's publication of superintelligence to try and get the world to pay attention to these problems in advance and prepare. I think I disagree with Eliezer that that has been on the whole bad. In some important ways the situation is looking a lot better than the alternative ways it could have been. I think it's important that you have several of the leading AI labs making not only significant lip service but also some investments in things like technical alignment research, providing significant public support for the idea that the risks of truly apocalyptic disasters are real. I think the fact that the leaders of OpenAI, Deep Mind and Anthropic all make that point. They were recently all invited along with other tech CEOs to the White House to discuss AI regulation. You could tell an alternative story where a larger share of the leading companies in AI are led by people who take a completely dismissive, denialist view and you see some companies that do have a stance more like that today. So a world where several of the leading companies are making meaningful efforts and you can do a lot to criticize could they be doing more and better and would have been the negative effects of some of the things they've done but compared to a world where even though AI would be reaching where it's going a few years later, those seem like significant benefits. And if you didn't have this kind of public communication you would have had fewer people going into things like AI policy, AI alignment research by this point and it would be harder to mobilize these resources to try and address the problem when AI would eventually be developed not that much later proportionately. I don't know that attempting to have public discussion understanding has been a disaster. I have been reluctant in the past to discuss some of the aspects of intelligence explosion, things like the concrete details of AI takeover before because of concern about this problem where people who only see the international relations aspects and zero sum and negative sum competition and not enough attention to the mutual destruction and senseless deadweight loss from that kind of conflict. At this point we seem close compared to what I would have thought a decade or so ago to these kinds of really advanced AI capabilities. They are pretty central in policy discussion and becoming more so. The opportunity to delay understanding and whatnot, there's a question of — For what? I think there were gains of building the AI alignment field, building various kinds of support and understanding for action. unity to delay understanding and whatnot, there's a question of — For what? I think there were gains of building the AI alignment field, building various kinds of support and understanding for action. Those had real value and some additional delay could have given more time for that but from where we are, at some point I think it's absolutely essential that governments get together at least to restrict disastrous reckless compromising of some of the safety and alignment issues as we go into the intelligence explosion. Moving the locus of the collective action problem from numerous profit oriented companies acting against one another's interest by compromising safety to some governments and large international coalitions of governments who can set common rules and common safety standards puts us into a much better situation. That requires a broader understanding of the strategic situation and the position they'll be in. If we try and remain quiet about the problem they're actually going to be facing it can result in a lot of confusion. For example the potential military applications of advanced AI are going to be one of the factors that is pulling political leaders to do the thing that will result in their own destruction and the overthrow of their governments. If we characterize it as things will just be a matter of — you lose chatbots and some minor things that no one cares about and in exchange you avoid any risk of the world ending catastrophe, I think that picture leads to a misunderstanding and it won't make people think that you need less in the way of preparation of things like alignment so you can actually navigate the thing, verifiability for international agreements, or things to have enough breathing room to have caution and slow down. Not necessarily right now, although that could be valuable, but when it's so important when you have AI that is approaching the ability to really automate AI research and things would otherwise be proceeding absurdly fast, far faster than we can handle and far faster than we should want. So yeah, at this point I'm moving towards sharing my model of the world to try and get people to understand and do the right thing. There's some evidence of progress on that front. Things like the statements and movements by Geoff Hinton are inspiring. Some of the engagement by political figures is reason for optimism relative to worse alternatives that could have been. And yes, the contrary view is present. It's all about geopolitical competition, never hold back a technological advance and in general, I love many technological advances that people I think are unreasonably down on, nuclear power, genetically modified crops. Bioweapons and AGI capable of destroying human civilization are really my two exceptions and yeah we've got to deal with these issues and the path that I see to handling them successfully involves key policymakers and the expert communities and the public and electorate grokking the situation therein and responding appropriately.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.