Noam Shazeer: belief

Transcript context

…Well, I would say that the pivot to hardware oriented around that was an important transition, because before that, we had CPUs and GPUs that were not especially well-suited for deep learning. And then we started to build TPUs at Google that were really just reduced-precision linear algebra machines, and then once you have that then you want to exploit it. It seems like it's all about identifying opportunity costs. Like, okay, this is something like Larry Page, I think, used to always say: "Our second biggest cost is taxes, and our biggest cost is opportunity costs. " If he didn't say that, then I've been misquoting him for years. But basically it’s like, what is the opportunity that you have that you're missing out on? In this case, I guess it was that you've got all of this chip area, and you're putting a very small number of arithmetic units on it. Fill the thing up with arithmetic units! You could have orders of magnitude more arithmetic getting done. Now, what else has to change? Okay, the algorithms and the data flow and everything else. And, oh, by the way, the arithmetic can be really low precision, so then you can squeeze even more multiplier units in.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.