Eliezer Yudkowsky: belief

Transcript context

…We briefly covered this, but I think this is an important topic, so I want to get the explanation again of why are you pessimistic that once we have these human level AIs, we’ll be able to use them to work on alignment itself? I think we started talking about whether verification is actually easier than generation when it comes to alignment, Yeah, I think that’s the core of it. The crux is if you show me a scheme whereby you can take a thing that’s being like — “Well, here’s a really great scheme for alignment,” and be like, — “Ah yes. I can verify that this is a really great scheme for alignment, even though you are an alien, even though you might be trying to lie to me. Now that I have this in hand, I can verify this is totally a great scheme for alignment, and if we do what you say, the superintelligence will totally not kill us.” That’s the crux of it. I don’t think you can even upvote downvote very well on that sort of thing. I think if you upvote-downvote, it learns to exploit the human readers. Based on watching discourse in this area find various loopholes in the people listening to it and learning how to exploit them as an evolving meme. Yeah, well, the fact is that we can just see how they go wrong, right?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.