Dwarkesh Patel: uncertainty

Transcript context

…And most humans don't have a consistent set of morals to begin with, right? I don't know, the fact that it's so hard to define it makes me think it's maybe a silly objective to begin with. Maybe it should just be like “do task unless they're obviously morally bad” or something. Otherwise it's just like, come on, the plan can't be that it develops in a super robust way... Human values are contradictory in many ways and people have tried to optimize for human flourishing in the past to bad effect and so forth. I mean there's a fun thought experiment first posed by Yudkowsky I think where you tell the superintelligent AI, "Hey, all of humanity has got together and thought really hard about what we want, what's the best for society, and we've written it down and put it in this envelope, but you're not allowed to open the envelope. But do what's in the envelope.” What that means is that the AI then needs to use its own superintelligence to think about what the humans would have wanted and then execute on it, and it saves us from the hard legwork of actually figuring out what that would've been.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.