Evidence receipt / prediction
Published · transcript-backedShawn Wang: prediction
3 Sept 2024 Latent Space Efficiency is Coming: 3000x Faster, Cheaper, Better AI Inference from Hardware Improvements, Quantization, and Synthetic Data Distillation
“So when you, when you quantize things, obviously you're going to lose precision because you just have less bits to store information in.”
Source trail
Everything needed to verify it.
- Speaker
- Shawn Wang
- Attribution
- Verified speaker
- Claim type
- prediction
- Recorded
- 3 Sept 2024
- Publisher
- Latent Space
Transcript context
…What do you mean by that? So when you, when you quantize things, obviously you're going to lose precision because you just have less bits to store information in. My sense is that when you quantize in vision, you can preserve the, maybe like the most, the principal components of features. More accurately, and that's actually what you really care about, whereas in language, you have a lot of complex interplay between meanings of words that, uh, you know, Anthropic calls it superposition, maybe. And when you quantize things, you might lose the lower precision bits, which actually matter a lot in language compared to vision. I don't know if you have any perspective on the precision trade off. I would need to talk to experts about this, but my intuition has been that The smaller the model, the more the weight matters. So what do I mean by that? So if the model is very small, you have very few parameters. So those parameters, like the information that they transmit needs to be more precise. So my intuition has been that, for example, at ResNet 18, when we would do quantization and we didn't do quantization, our training after that, it would just completely fall off a precipice. And that was something that we needed to be extremely careful on. And that's why there are so many techniques that were designed for that. But that is my personal intuition that I developed and with large language models, given that they are so large, small changes may impact them less than in the case of a very, very small computer vision model, obviously that falls apart with like the large, Computer vision models, like segment anything or things like that. But if you have a very small single task, ResNet 18, if you lose a little bit your weights and don't quantize it the right way, your results all of a sudden are going to like go completely bollocks very fast.…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.