Nathan Labenz: evaluation

Transcript context

…h noise is there? And also I'm kind of wondering this might, you know, AI might be the perfect technology for this underlying computing substrate in the sense that it's quite tolerant of noise, right? We have models that get quantized and we have, you know, token distribution logic outputs that if they're slightly off, you know, more often than not, it doesn't even change the token that you're going to see. So I'm kind of wondering like you've got all those layers that can make this work. How much is dialing in the core unit to a higher level of precision and how much is sort of accepting that a little bit of noise is OK as it propagates through? That's a fantastic question. And in fact, there's been a pretty broad body of knowledge and my group has been quite involved in this over the last, I don't know, 10 or 15 years, which really made the following observation. It said, hey, listen, you know, the applications that we're trying to run at the highest level, they're tolerant to noise. They're statistical applications and noise is a natural thing. The underlying substrate with which we're trying to, you know, sort of do the computations and and run these applications is therefore it's reasonable that it should be noisy. It turns out that in practice that is, I think on a high level it's a very reasonable thing to think in practice. It's a very challenging thing to make work in the practical systems we want to build and the practical levels of skills we want. And the reason for that is that way down over here at the physics of how you do computation versus way up over here at the level of the applications you're interested in, there are many layers of abstraction in between. And the way that we go from the complexity, right, of a single transistor scaled up to what multiple 100 billion transistors and all of the software that needs to run that is really dependent on the integrity of those abstractions. Now, the problem is today we don't know how to build abstractions in a robust and scalable way that, you know, sort of represent the noise of that underlying substrate. And so there ends up being a disconnect between the noise and how it's represented at the, you know, lowest physical level versus the way that it's represented at that application level. And so while, you know, we talk about there being noise tolerances and, and AI models are tolerant and that they are, right, We're able to do things like quantization, as you pointed out, but really we're doing those kinds of things with very carefully represented noise sources, right? are tolerant and that they are, right, We're able to do things like quantization, as you pointed out, but really we're doing those kinds of things with very carefully represented noise sources, right? Quantization noise is a very carefully represented noise source and one which you need to be able to properly represent throughout your layers of abstraction and digital. You know, quantization is something that we do have ways of building robust abstractions for, but this analogue noise is one that that we really don't. And that's why essentially what you do, you know, when you do digital compute is you say, hey, listen, there's all sorts of noise that analog might lead to. But we drown all of those out by thinking about, you know, the signals being a 0 or A1. So that that's the dominant source of noise. And that's all that I now have to represent over my layers of abstraction. Everything else basically doesn't matter. But because this is a very well represented form of noise, quantization noise, I know how to deal with it at the algorithmic level. And I can apply all my algorithmic techniques. And that's what the industry has done very successfully. But now as you want to sort of leverage analogue, you still need those levels of abstraction. That's the key to achieving systems at scale and and you know, systems that you can build architectural abstractions and the software abstractions on top of. So I would say that the need to be accurate and precise is still, you know, kind of brutally high. And so that's that's very, very important. Now you also ask the question of OK, then how precise is your is your approach? Because now you know, I'm telling you we need to know that and understand that very well. And it turns out that the dominant source of noise that we have in our approach based on these capacitors, there could be many sources. There's the electronic, you know, the discrete nature of electronic charge causes noise and things like that. But it actually turns out the dominant sources, the variability of the capacitors that we we can fabricate on a chip. Now, it turns out that those capacitors are really critically dependent on geometric properties. So basically the distance between 2 metal wires. And it turns out that geometry is really the one thing we can control very well in CMOS processes. It turns out we use this processing approach called lithography that gives us very precise geometric control. That's the reason we can build, you know, 532 nanometer transistors. Turns out we don't need anywhere near that precision for the capacitors that we use. But, but it's really because of this alignment with this geometric control that this particular approach has that allows it to be brutally accurate in the ways that you need it to be through all of these layers of abstraction to be able to scale up. We've measured these things in a lot of detail. It turns out for the kinds of capacitors we use, you see variations that are on the order of, you know, sort of 10 parts per million, right. o scale up. We've measured these things in a lot of detail. It turns out for the kinds of capacitors we use, you see variations that are on the order of, you know, sort of 10 parts per million, right. So giving you levels of precision that are in the neighborhood of 20 bits of precision, which it turns out is well beyond what we need for the quantization kinds of, you know, levels that we care about, which are typically at the level of eight bits and you know, higher than that in some cases. So we've, we've had to characterize these things very carefully because the noise does matter as you're trying to build these abstractions all the way up. And that's the level of precision we've we've gotten here, which is what made, which made this approach so practical and where we've been able to now scale it and demonstrate it across, you know, all sorts of chips and systems. I note that you have spent quite a bit of time getting a neural net onto one of your chips and onto I think a laptop like a edge edge device, right? Trying to get into edge devices which are more sensitive to power consumption and more sensitive to, you know, I you, you just don't have the affordances that you have in a data center, so to speak. So what would be the comparison be like, you know, if you were to use a normal GPU versus A1 of your chips, What is a comparison on like, let's say energy savings or like how do you compare the 2? Yeah, that's a great question. And and I think it really points to the fact that if you want to now leverage this, you know, fundamentally new technology, analogue, it's not enough to just build that technology and make it robust. You end up having to build the entire architecture and the entire software around, you know, harnessing and extracting it's full efficiency. And the reason I say that is I can give you 2 answers. One is at the level of the core technology, the thing that this analogue computing engine does, what level of efficiencies do we have?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.