David Deutsch: preference

Transcript context

…And then why is fun central to this instead of another emotion? So, like, for example, Aristotle thought that a widely defined sense of happiness is “what should be the goal of our endeavors?” Why fun instead of something like that? Well, that's defining it vaguely enough. The point is, the underlying thing is as far as going one level below, we really understand that we'd need to go about seven levels below that, which we can't do yet. But the important thing is that there are several kinds of knowledge in our brains. And the one that is written down in the exercise book that says you should do this number of reps. And you should power through this. And it doesn't matter if you feel that and so on. That's an explicit theory. And it contains some knowledge, but it also contains errors. All our knowledge is like that. We also have other knowledge, which is contained in our biology, it's contained in our genes. We have knowledge that is inexplicit, like our knowledge of grammar is always my favorite example, as we know why certain sentences are acceptable and why they're unacceptable. But we can't state explicitly all in every case, why it isn't, or why it is. And then as there's so that there's explicit and explicit knowledge, there's conscious and unconscious knowledge. All those are bits of programming the brain, they’re ideas, they are they they are bits of knowledge in this, if you define knowledge as information with causal power, they are all information with causal power. They all contain truth, and they all contain error. And it's always a mistake, to shield something to just shield one of them from criticism or replacement. Not doing that is what I call the fun criterion. Now, you might say that that's a bad name, but it's the best I can find. So why would creating an AGI through evolution necessarily entail suffering? Because the way I see it, it seems to be your theory that you need to be a general intelligence in order to feel suffering, but by the point and evolution, simulated Bing is a general intelligence and we can just stop the simulation. So where's the suffering coming from?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.