Shawn Wang: evaluation

Transcript context

…I would need to talk to experts about this, but my intuition has been that The smaller the model, the more the weight matters. So what do I mean by that? So if the model is very small, you have very few parameters. So those parameters, like the information that they transmit needs to be more precise. So my intuition has been that, for example, at ResNet 18, when we would do quantization and we didn't do quantization, our training after that, it would just completely fall off a precipice. And that was something that we needed to be extremely careful on. And that's why there are so many techniques that were designed for that. But that is my personal intuition that I developed and with large language models, given that they are so large, small changes may impact them less than in the case of a very, very small computer vision model, obviously that falls apart with like the large, Computer vision models, like segment anything or things like that. But if you have a very small single task, ResNet 18, if you lose a little bit your weights and don't quantize it the right way, your results all of a sudden are going to like go completely bollocks very fast. I do agree with that intuition. I think one of the things that people are talking about now is like very extreme quantization. There is this paper on ternary models, the 1. 58 bit models. I don't know how much legs that is, but people seem to be reproducing it in open source. And it's something that a lot of people are talking about. I don't know what to make about it because I don't think it's adopted seriously by the large labs. Yeah, I'm not sure about that, but I do I think that in a way it's like with such a large model, you almost need just that directional number, like yes or no. And then it go, it's like almost like a gate of like this direction versus this direction. And because it has so many parameters, yes or no for those gates in a way matters more than the full exact precise number that we get there. Yeah. I like to think about it like in physics. We have come up with very precise weights for our bar, like constants, right? But those constants have determined to work in a lot of circumstances. Those have been very specific. For that specific equation. And it was like a lot of graph while in the super large model, it's more of like a directionality that matters than the full number of the way that would be my personal intuition, but there are extreme experts that have been working on quantization for many, many years that could answer that question better.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.