Alessio Fanelli: evaluation

Transcript context

…That's a very good question, and there's no good answer. There's so many parameters to take into account from the scaling loss, training time to get the best performance, the GPU constraint, and on what different hardwares, and we think about meta, but also of the community, and people are not just using 800, but there's 800, there's different size of GPUs memory. So which size will fit in what, and what is the most useful? Also at inference time, not just at fine tuning time, then you can maybe do some tricks at inference time to quantize it a bit, or FP16 or FP8 now. All those constraints makes it very, very challenging. At inference time, you have a lot of costs. So how to trade off between inference costs and training costs? It's a very challenging problem. In general, we tend to think, in particular for Llama 3, Llama 2 maybe I would say it's like Llama 1, we had a flagship model which was 70b, it's also because the project was taking some routes to reproducing Chinchilla, which was a 70b. For Llama 3, we also moved to one size more, the flagship model for 0.5b. I think there was also the question of, we want a model at this time, we have this amount of compute, given the scaling laws and the amount of tokens we have to train it. What would be the right balance to still fit in at inference time? So we try to have some trade-offs like that. Yeah. You mentioned Chinchilla is the best way to go, but then you tweeted recently, don't fall into the Chinchilla trap if you want your model to be used by billions of people. So what's the updated state of scaling loss? I think there was obviously the Kepler, and then there was Chinchilla, and then people kind of got the Llama scaling law, like the 100 to 200x parameter to token ratio. What's your updated thinking on how to think about scaling loss when you get model size and training data? Right. So, you know, as you said, this Kepler paper with scaling laws, but they figured out, basically they tried two dimensions, the model weights and the number of training time, like number of steps, training tokens, epochs. And for that, they figured that model size is what matters. So GPT-3 was way too big compared to the actual number of training tokens because they did a mistake, not adapting the scheduler. That's what Chinchilla emphasized and discovered. To be fair, I think OpenAI knew that at the time of Chinchilla paper, but yeah, basically Chinchilla said we have to revisit the scaling laws originally published by Kepler and emphasize much more the importance of training tokens. And they did like some really good scaling laws showing that there's an optimal, basically you need to double the number of training tokens every time you double the training weights to get an optimal ratio so that for a finite number of compute, you will end with the best results in your paper. And what I call the Chinchilla trap is that, that's good if you want the best flagship model that obtains the highest performance on your paper. But if you want to use your model at inference time, inference, the two dimensions, one remains the model weights, but one drops the number of tokens you train it, number of steps. And so to be compute efficient at inference time, it's much better to train it much longer training time, even if it's an effort, an additional effort, than to have a bigger model. That's what I call, I refer to the Chinchilla trap. Not that Chinchilla was wrong, but if you can see your inference time, you need to go beyond Chinchilla. And in fact, that's what Llama1 folks did by overtraining in the sense they could have get a better performance in paper, but they prefer to create the best artifact that will be used by the community.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.