Ben Gilbert: evaluation

Transcript context

……most people in the field thought this was not going to work, and most people in Google thought this was not going to work. And here’s a little bit on why. It’s a little technical, but follow me for a second. All the research from that period of time pointed to the idea that you needed to be synchronous. So all the compute needed to be really dense, happening on a single machine, with really high parallelism, like what GPUs do, that you really would want it all happening in one place. It’s really easy to go look up and see, hey, what are the computed values for everything else in the system before I take my next move? What Jeff Dean wrote with Distbelief was the opposite. It was distributed across a whole bunch of CPU cores and potentially all over a data center or maybe even in different data centers. In theory, this is really bad because it means you would need to be constantly waiting around on any given machine for the other machines to sync their updated parameters before you could proceed. But instead, the system actually worked asynchronously without bothering to go and get the latest parameters from other cores. You were updating parameters on stale data. You would think that wouldn’t work. The crazy thing is it did. Okay, so you’ve got Distbelief. What do they do with it now? They want to do some research. So they try out, can we do cool neural network stuff? And what they do in a paper that they submitted in 2011, right at the end of the year, is—and I’ll just give you the name of the paper first—Building High-Level Features Using Large-Scale Unsupervised Learning, but everyone just calls it the cat paper. The cat paper.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.