Daniel Kokotajlo: belief

Transcript context

…The other notable thing about your model is, you got this superhuman thing at the end of it and then it seems to just go through the tech tree of mirror life and nanobots and whatever crazy stuff. And maybe that part I’m also really skeptical of. If you look at the history of invention, it just seems like people are just trying different random stuff, often even before the theories about how that industry works or how the relevant machinery works is developed; like the steam engine was developed before the theory of thermodynamics, the Wright brothers seemed like they were just experimenting with airplanes, and is often influenced by breakthroughs in totally different fields. Which is why you have this pattern of parallel innovation, because the background level of tech is at a point at which you can do this experiment. Machine learning itself is a place where this happened, right? Where people had these ideas about how to do deep learning or something. But it just took a totally unrelated industry of gaming to make the relevant progress, to get the whole, basically the economy as a whole advanced enough that deep learning, Geoffrey Hinton’s ideas could work. So I know we’re accelerating way into the future here, but I want to get to this crux. So again, we have that three part division of the superhuman coder, then the complete AI researcher and then the super intelligent, you’re not jumping ahead to that one. So now we’re imagining systems that are true super intelligence, they are just better than the best humans at everything, including being better at data efficiency and better at learning on the job and stuff like that. Now, our scenario does depict a world in which they’re bottlenecked on real world experience and that sort of thing. I think that if you want a contrast, some people in the past have proposed much faster scenarios where they email some cloud lab and start building nanotech right away by just using their brains to figure out appropriate protein folding and stuff like that. We are not depicting that in our scenario. In our scenario, they are in fact bottlenecked on lots of real world experience to build these actual practical technologies, but the way they get that is they just actually get that experience and it happens faster than humans would. And the way they do that is they’re already super intelligent, they’re already buddy-buddy with the government, the government deploys them heavily in order to beat China and so forth, and so all these existing US companies and factories and military procurement providers and so forth are all chatting with the superintelligences and taking orders from them about how to build the new widget and test it, and they’re downloading super intelligent designs and manufacturing them and then testing them and so forth. And then the question is, they are getting this experience, they’re learning on the job, quantitatively, how fast does this go? Is it taking years or is it taking months or is it taking days? In our story, it takes about a year and we’re uncertain about this. Maybe it’s going to take several years, maybe it’s going to take less than a year. Here are some factors to consider for why it’s plausible that it could take a year: One, you’re going to have something like a million of them. And quantitatively that’s comparable in size to the existing scientific industry. I would say, like maybe it’s a bit smaller, but it’s not dramatically smaller. Two, they’re thinking a lot faster. They’re thinking like 50 times speed or like 100 times speed that I think counts for a lot. And then three, which is the biggest thing, they’re just qualitatively better as well. So not only are there lots of them and they’re thinking very fast, but they are better at learning from each experiment than the best human would be at learning from that experience. Yeah, I think the fact that there’s a million of them or the fact that they’re comparable to maybe the size of this key researcher population of the world or something. I think there’s more than a million researchers in the world, but……

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.