Shawn Wang: belief

Transcript context

…Um, so yeah. Useful. Uh, cool. Uh, I think. There, there’s more general, like broad questions, but like, I guess what, what do you, uh, wish you were asked more in, in, in general, like, you know, like you, you have such a broad scope. We’ve covered the hardware, we’ve covered the, the, the models research. Yeah. I mean, I think, uh, one thing that’s kind of interesting is, you know, I, I did a undergrad thesis on neural network, uh, training, uh, uh, parallel neural network training, uh, back in 1990 when I got exposed to neural nets and I always felt kind of, they were the right abstraction. Uh, but we just needed way more compute than we had then. Mm-hmm. So like the 32 processors in the department parallel computer, you know, could get you a, a little bit more interesting, uh, model, but not, not enough to solve real problems. And so starting in 2008 or nine, you know, the world started to have enough computing power through Moore’s law and, you know, larger, interesting data sets to train on to actually, you know, start training neural nets that could tackle real problems that people cared about. Yeah. Speech recognition. Vision, and eventually, uh, language. Um, and so, um, when I started working on neural nets at Google in, in late 2011, um, you know, I really just felt like we should scale up the size of neural networks we can train using, you know, large amounts of parallel computation. And so, uh, I actually, uh, revived some ideas from my undergrad thesis where I’d done both model parallel and data parallel, uh, training and I compared them. Uh, I, I called them. I’ve been doing this since I was eight. It was something different. There was like pattern partitioned and, you know, model partitioned or something.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.