Public evidence record
Published podcast speaker
Claim ledger
50 transcript-backed records
01 / evaluation
“It hurts your throughput, in fact, because the throughput of your chip is the product of how much you get done per clock cycle—which is based on area efficiency—times how many clocks you get per second.”
02 / evaluation
“The trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs $30 million because it requires an entire tape-out.”
03 / recommendation
“Instead of forming a running sum of everything that comes in, I will actually have two different running sums.”
04 / evaluation
“I have this cost, which is 3 x n x p AND gates over here, compared to p x q gates in the actual circuit that is doing the thing I care about.”
05 / evaluation
“We don’t want to bring this full matrix in from the register file every cycle, because that would cost too much in terms of wiring from the register file.”
06 / preference
“Really there’s only one addition per dot product, but we like to start with zero.”
07 / evaluation
“If you split it in the middle, you can hit twice the clock frequency. That’s great, you get twice the performance, but at the cost of an extra register, which means more storage.”
08 / evaluation
“I think the big observation you’ve made is that there’s this quadratic scaling with bit width, which is very effective and is the single reason low-precision arithmetic has worked so well for neural nets.”
09 / evaluation
“Generally, people will go a little bit larger than this. They don’t really want to be exactly at the balance point because real-world efficiencies aren’t as good as a roofline analysis would say.”
10 / evaluation
“It is simply that I need to read all of my total parameters from memory into the chips, and that takes a certain amount of time. If I use all of my memory bandwidth, I can’t do any better than that.”
11 / belief
“I don’t have that information, but I think a good ballpark is a 33% split between each of them.”
12 / belief
“I think that’s a pretty good assumption. If you look at the numbers, it might also turn out that it’s one tier down, and it’s DDR versus flash.”
13 / commitment
“I can motivate the batch at least a little bit. We will see exactly why batch is such a favorable optimization.”
14 / belief
“I think if you do it with a spreadsheet and actually model it out, you might notice when the money’s going down the drain.”
15 / belief
“I think often the number of active parameters could be in the range of a hundred billion, something like that.”
16 / belief
“I think starting with equalizing in cost is right, but depending on how you model the cost, this comes close to equalizing in data.”
17 / belief
“I think we just have to make some real-world assumptions here in order to do that.”
18 / commitment
“We will directly jump to the ratio of the time spent on scale-up over the time spent on scale-out.”
19 / belief
“If we think that the memory in a rack is a bottleneck, then there’s a constraint on how fast we can go.”
20 / recommendation
“If you were designing your hardware, you could say, “I didn’t need that much memory because I don’t need the weights to fit in one rack.”
21 / evaluation
“I think that indicates that this is the reasonably balanced cost point, and going massively beyond that would be cost-prohibitive.”
22 / evaluation
“I can’t use different pipeline stages in parallel because they’re not running at the same time, but I can use all the GPUs in my scale-up domain in parallel to load the weights.”
23 / evaluation
“I’m kind of shocked to see spinning disk being used at all, because it’s such an old technology.”
24 / evaluation
“Because of the ability to do pipelining, a rack at least should not be a constraint on your ability to fit the model parameters.”
25 / evaluation
“If I think about what that means, it means I actually have time to read the HBM twice.”
26 / preference
“What that means is, given that I know I’m going to be holding something for five minutes, I would like to pick a memory that I can read every five minutes.”
27 / evaluation
“If I think about what cost means… To run this inference, I’m going to use the GPU for a certain number of seconds, like one millisecond or 20 milliseconds.”
28 / evaluation
“A way to think of this is that if I have too many of these things sitting in my HBM, if I fill up my HBM with just KV caches that I’m not using, I can’t use that GPU.”
29 / belief
“The product we aim to build is far ahead on throughput, but then, actually, the surprising thing is we're competitive with the best on latency as well. I think that is a unique thing in offering both in the same place.”
30 / uncertainty
“I don't know. The cost of a fab is extremely expensive. I recognize that also the cost of a lab is extremely expensive, too.”
31 / belief
“Our product is both training and inference, but I think the first sales will be an inference.”
32 / belief
“There's a few aspects to that. I think the first one is pick your memory system right.”
33 / belief
“You typically, I think the average rates really depends on process and so on, but small single digit number of defects per chip.”
34 / belief
“I think hardware companies in general are really exciting because you have such a broad range of skills of people on the team.”
35 / belief
“Data centers, which I think is power, primarily a little bit of buildout, but primarily power and good infrastructure there.”
36 / belief
“If you say they're going to service it never, then I think you actually can get where you want to be, but maybe with 100% tax on reliability rather than 10%.”
37 / belief
“The Frontier Labs have done a pretty good job of exploring it, but I think, as the hardware changes, the shape of the model should change for sure.”
38 / belief
“There's what the Rust marketing will say, which is "safe without a garbage collection," which I think is a real… It is the objective thing that you can say is different, but sort of buries the lead, which is also just it's got nice type system features that Go doesn't have.”
39 / belief
“We think probably the main thing will be similar to where NVIDIA is at, which is 4-bit precision. But I think a mix of different precisions is useful for just when you look at the research, sometimes you want some layers in higher precision or lower precision, and so on.”
40 / belief
“I think so. I'll just comment, by the way, this is a great time to be a supplier in this place.”
41 / belief
“Whereas I think a company like Google or NVIDIA, the next chip has to work for sure.”
42 / belief
“More labs, I think, is still interesting. Just can we do more on model architecture is always interesting.”
43 / evaluation
“I think there's ultimately… At least for existing chips in the market, there's a really uncomfortable trade-off between latency and throughput.”
44 / prediction
“Parameter count should grow much faster than context length actually, just because of the underlying physics of what's available.”
45 / evaluation
“I think that's a pretty good fit because you have different parts of your company that are skilled at different areas, and it allows you to keep both of them occupied without having instead every 2 years doing a massive risk release.”
46 / evaluation
“What that means for us is that ecosystem already exists, and that way of operating, where you say, "I'm just going to staff a 50-person team to write software for this chip," works really well if you're trying to sell to frontier labs.”
47 / evaluation
“" Because the performance of software running on CPUs is really bad, but running on GPUs, or TPUs, or AI chips in general, actually that is the main focus.”
48 / evaluation
“There's this really cool technique called 'cuckoo hashing' where you hash into two different locations, and then you use the bucket which is less full. It's been in the literature for decades and yet the best hash table implementations don't use it because it's somehow not practical.”
49 / evaluation
“I think most ML researchers, especially of the time, were not super deep in what hardware wants, and what is… Mechanical sympathy is sometimes a term that's used for that.”
50 / evaluation
“More recent TPUs and more recent AI chips in general can't do that because the market has moved, and the table stakes are much higher.”