Reiner Pope: evaluation

Transcript context

…But is there some hand-wavy story you can tell about parallelization where both Transformers and TPUs are about really internalizing the importance of parallelization? Definitely, I put it somewhat on people, actually. It is just true. Hardware is massively parallel. You've got tens of billions, hundreds of billions of transistors on your chip, and it takes maybe 100 clock cycles to get from one side of the chip to the other. You can't do a sequential computation involving transistors on both sides of the chip. The hardware is just fundamentally parallel, and you have to take advantage of that. TPU v1 and all later TPUs naturally took advantage of that. Matrix Multiply is really nice because it is so parallel. I think on the hardware side that's generally understood. I think most ML researchers, especially of the time, were not super deep in what hardware wants, and what is… Mechanical sympathy is sometimes a term that's used for that. What is the term? It kind of—…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.