Ilya Sutskever: prediction

Transcript context

…Isn't it odd that we have the data we needed exactly at the same time as we have the transformer at the exact same time that we have these GPUs? Like is it odd to you that all these things happened at the same time or do you not see it that way? It is definitely an interesting situation that is the case. I will say that it is odd and it is less odd on some level. Here's why it's less odd — what is the driving force behind the fact that the data exists, that the GPUs exist, and that the transformers exist? The data exists because computers became better and cheaper, we've got smaller and smaller transistors. And suddenly, at some point, it became economical for every person to have a personal computer. Once everyone has a personal computer, you really want to connect them to the network, you get the internet. Once you have the internet, you suddenly have data appearing in great quantities. The GPUs were improving concurrently because you have smaller and smaller transistors and you're looking for things to do with them. Gaming turned out to be a thing that you could do. And then at some point, Nvidia said — the gaming GPU, I might turn it into a general purpose GPU computer, maybe someone will find it useful. It turns out it's good for neural nets. It could have been the case that maybe the GPU would have arrived five years later, ten years later. Let's suppose gaming wasn't the thing. It's kind of hard to imagine, what does it mean if gaming isn't a thing? But maybe there was a counterfactual world where GPUs arrived five years after the data or five years before the data, in which case maybe things wouldn’t have been as ready to go as they are now. But that's the picture which I imagine. All this progress in all these dimensions is very intertwined. It's not a coincidence. You don't get to pick and choose in which dimensions things improve. How inevitable is this kind of progress? Let's say you and Geoffrey Hinton and a few other pioneers were never born. Does the deep learning revolution happen around the same time? How much is it delayed?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.