Dwarkesh Patel: preference

Transcript context

…This shows up all the way up and down the stack. This is close to the bottom, to the gates. There’s a version that’s maybe even closer to the gates in the precision of the number format that you choose to use. We saw that same effect. There’s a squared versus linear term going on both purely in the precision of the ALU, but also in the size of the matrix. This unit is the next bigger unit. We had the multiplication circuit, and on top of that we have a pretty large systolic array. I drew it as 2x2, but older TPUs were described as 128x128 of this circuit shown here. This ends up being the most efficient known circuit for implementing a matrix multiply. We’ve talked about how it seems obvious that you should try to maximize compute relative to communication. What are non-obvious trade-offs that keep you up at night, about whether you should do X or Y and it’s not obvious what the answer is? Most of the decisions in chip design are sizing decisions. Already in what we’ve drawn so far… AI chips all have this circuit in them. They have a systolic array, and somewhere near it a register file providing inputs and outputs. Even within this scope, the sizing questions you have are: how big should I make my systolic array, and how big should I make the register file? These two questions are coupled. One way to think of it is to set a budget for what percentage of chip area you want to spend on data movement. Maybe I say that I want this to be 10% and the systolic array to be 90%. Then I can size my register file. Bigger register files are more flexible. They allow me to get more application-level performance out, but they take away from the area spent on the systolic array.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.