Nathan Labenz: evaluation

Transcript context

…l signals to now this point where it's accessing analogue signals, where you're doing compute internally that's generating not just zeros and ones, but, you know, much more richly represented signals. And the, the big problem in our thinking was how do we take this approach that works well for zeros and ones and skill it to this new regime where we need much more than zeros and ones and therefore much higher precision. And much of the community and, and our initial work to try to explore and understand this really said, let's take the approaches that we've used for traditional memory and try to scale them to this new regime. And that doesn't work. I think the big transformation was to say, hey, listen, this is, this looks less like a memory design problem where you're trying to access zeros and ones. It looks much more like a very high precision analogue design problem. Well, it turns out that there's been a lot of really important research and and work that has led to extraordinarily precise analog circuits. We've built 20 bit ABCS. You can buy these from companies like Analog Devices and Texas Instruments. Think about high reliability applications like medical and aerospace and automotive. So the big breakthrough was really to say, OK, let's take those approaches which have not traditionally been used or considered for memory design and bring them into this architecture of memory design and in memory computing to enable that architecture to scale to this new regime of, you know, needing this level of precision. And that led us to this approach called switched capacitor in memory computing, where this technique switched capacitors has been used and used robustly as I mentioned for extreme precision analog to digital converters. Our innovation was to figure out how to use it in an architecture that now does in memory computing for AI. It really sounds like building on a combinatorial kind of like, you know, innovation from other, other sectors perhaps, and, you know, putting that together in order to be used for compute. Yeah. And that's the privilege that, you know, we have as as fundamental researchers, right, where we're not just tied to a particular problem. And we think, you know, kind of on a fundamental level, when you do that, you don't get siloed into the approaches that have been adopted and that are kind of the prevailing techniques for a certain, you know, certain problem and, and how you solve it. You can think broadly. And, and that really is a privilege. And it's, you know, that perspective that we were able to leverage to, you know, really drive what I believe is a critical breakthrough in enabling robust and scalable analog compute to unlock this efficiency for for AI. Can you talk about at the lowest level how precise these things are? I mean, if it we are used to doing in zeros and ones, right? Like how many bits or how many significant digits do your kind of core units operate at? How much noise is there? And also I'm kind of wondering this might, you know, AI might be the perfect technology for this underlying computing substrate in the sense that it's quite tolerant of noise, right? h noise is there? And also I'm kind of wondering this might, you know, AI might be the perfect technology for this underlying computing substrate in the sense that it's quite tolerant of noise, right? We have models that get quantized and we have, you know, token distribution logic outputs that if they're slightly off, you know, more often than not, it doesn't even change the token that you're going to see. So I'm kind of wondering like you've got all those layers that can make this work. How much is dialing in the core unit to a higher level of precision and how much is sort of accepting that a little bit of noise is OK as it propagates through? That's a fantastic question. And in fact, there's been a pretty broad body of knowledge and my group has been quite involved in this over the last, I don't know, 10 or 15 years, which really made the following observation. It said, hey, listen, you know, the applications that we're trying to run at the highest level, they're tolerant to noise. They're statistical applications and noise is a natural thing. The underlying substrate with which we're trying to, you know, sort of do the computations and and run these applications is therefore it's reasonable that it should be noisy. It turns out that in practice that is, I think on a high level it's a very reasonable thing to think in practice. It's a very challenging thing to make work in the practical systems we want to build and the practical levels of skills we want. And the reason for that is that way down over here at the physics of how you do computation versus way up over here at the level of the applications you're interested in, there are many layers of abstraction in between. And the way that we go from the complexity, right, of a single transistor scaled up to what multiple 100 billion transistors and all of the software that needs to run that is really dependent on the integrity of those abstractions. Now, the problem is today we don't know how to build abstractions in a robust and scalable way that, you know, sort of represent the noise of that underlying substrate. And so there ends up being a disconnect between the noise and how it's represented at the, you know, lowest physical level versus the way that it's represented at that application level. And so while, you know, we talk about there being noise tolerances and, and AI models are tolerant and that they are, right, We're able to do things like quantization, as you pointed out, but really we're doing those kinds of things with very carefully represented noise sources, right? are tolerant and that they are, right, We're able to do things like quantization, as you pointed out, but really we're doing those kinds of things with very carefully represented noise sources, right? Quantization noise is a very carefully represented noise source and one which you need to be able to properly represent throughout your layers of abstraction and digital. You know, quantization is something that we do have ways of building robust abstractions for, but this analogue noise is one that that we really don't. And that's why essentially what you do, you know, when you do digital compute is you say, hey, listen, there's all sorts of noise that analog might lead to. But we drown all of those out by thinking about, you know, the signals being a 0 or A1. So that that's the dominant source of noise. And that's all that I now have to represent over my layers of abstraction. Everything else basically doesn't matter. But because this is a very well represented form of noise, quantization noise, I know how to deal with it at the algorithmic level. And I can apply all my algorithmic techniques. And that's what the industry has done very successfully. But now as you want to sort of leverage analogue, you still need those levels of abstraction. That's the key to achieving systems at scale and and you know, systems that you can build architectural abstractions and the software abstractions on top of. So I would say that the need to be accurate and precise is still, you know, kind of brutally high. And so that's that's very, very important. Now you also ask the question of OK, then how precise is your is your approach? Because now you know, I'm telling you we need to know that and understand that very well. And it turns out that the dominant source of noise that we have in our approach based on these capacitors, there could be many sources. There's the electronic, you know, the discrete nature of electronic charge causes noise and things like that. But it actually turns out the dominant sources, the variability of the capacitors that we we can fabricate on a chip. Now, it turns out that those capacitors are really critically dependent on geometric properties. So basically the distance between 2 metal wires. And it turns out that geometry is really the one thing we can control very well in CMOS processes. It turns out we use this processing approach called lithography that gives us very precise geometric control. That's the reason we can build, you know, 532 nanometer transistors. Turns out we don't need anywhere near that precision for the capacitors that we use. But, but it's really because of this alignment with this geometric control that this particular approach has that allows it to be brutally accurate in the ways that you need it to be through all of these layers of abstraction to be able to scale up. We've measured these things in a lot of detail. It turns out for the kinds of capacitors we use, you see variations that are on the order of, you know, sort of 10 parts per million, right.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.