Public evidence record
Published podcast speaker
Claim ledger
22 transcript-backed records
01 / evaluation
“It hurts your throughput, in fact, because the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.”
02 / evaluation
“The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out.”
03 / evaluation
“I have this cost, which is 3 x n x p AND gates over here, compared to p x q gates in the actual circuit that is doing the thing I care about.”
04 / evaluation
“We don’t want to bring this full matrix in from the register file every cycle, because that would cost too much in terms of wiring from the register file.”
05 / evaluation
“If you split it in the middle, you can hit twice the clock frequency. That’s great, you get twice the performance, but at the cost of an extra register, which means more storage.”
06 / evaluation
“I think the big observation you’ve made is that there’s this quadratic scaling with bit width, which is very effective and is the single reason low-precision arithmetic has worked so well for neural nets.”
07 / evaluation
“Generally, people will go a little bit larger than this. They don’t really want to be exactly at the balance point because real-world efficiencies aren’t as good as a roofline analysis would say.”
08 / evaluation
“It is simply that I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.”
09 / evaluation
“I think that indicates that this is the reasonably balanced cost point, and going massively beyond that would be cost-prohibitive.”
10 / evaluation
“I can’t use different pipeline stages in parallel because they’re not running at the same time, but I can use all the GPUs in my scale-up domain in parallel to load the weights.”
11 / evaluation
“I’m kind of shocked to see spinning disk being used at all, because it’s such an old technology.”
12 / evaluation
“Because of the ability to do pipelining, a rack at least should not be a constraint on your ability to fit the model parameters.”
13 / evaluation
“If I think about what that means, it means I actually have time to read the HBM twice.”
14 / evaluation
“If I think about what cost means… To run this inference, I’m going to use the GPU for a certain number of seconds, like one millisecond or 20 milliseconds.”
15 / evaluation
“A way to think of this is that if I have too many of these things sitting in my HBM, if I fill up my HBM with just KV caches that I’m not using, I can’t use that GPU.”
16 / evaluation
“I think there's ultimately… At least for existing chips in the market, there's a really uncomfortable trade-off between latency and throughput.”
17 / evaluation
“I think that's a pretty good fit because you have different parts of your company that are skilled at different areas, and it allows you to keep both of them occupied without having instead every 2 years doing a massive risk release.”
18 / evaluation
“What that means for us is that ecosystem already exists, and that way of operating, where you say, "I'm just going to staff a 50-person team to write software for this chip," works really well if you're trying to sell to frontier labs.”
19 / evaluation
“" Because the performance of software running on CPUs is really bad, but running on GPUs, or TPUs, or AI chips in general, actually that is the main focus.”
20 / evaluation
“There's this really cool technique called 'cuckoo hashing' where you hash into two different locations, and then you use the bucket which is less full. It's been in the literature for decades and yet the best hash table implementations don't use it because it's somehow not practical.”
21 / evaluation
“I think most ML researchers, especially of the time, were not super deep in what hardware wants, and what is… Mechanical sympathy is sometimes a term that's used for that.”
22 / evaluation
“More recent TPUs and more recent AI chips in general can't do that because the market has moved, and the table stakes are much higher.”