Alessio Fanelli: prediction

Transcript context

…There's not like a lot of changes in terms of architectures. I think we can do a lot better in the future and not just like with transformers, but for instance, to me, like it doesn't make sense to use the same amount of compute per token for every token. Like there's architecture lack of flexibilities. There's a lot of research to go there, but still that's the best thing we have for now. And so it's the same recipe than in terms of architectures and training than Llama 2, but we put so much effort on scaling the data and the quality of data. There's now 15 trillion tokens compared to 2 trillion. So it's another venture there as well, including for the smaller models. One of the things I noticed on the paper is that you use Llama 2 to do the data cleaning for what went into Llama 3. I think there's a lot of chatter obviously about synthetic data and like there was the Rephrase the Web paper that came out maybe a few months ago about using, you know, Mastral to make training data better. Any learnings from that? It's like, is there, how much can you rewrite with the models? Like I'm sure people would love to hear more about it. Right. So it's very interesting, the research direction. Synthetic data in general, synthetic data for pre-training. My intuition is that the web is full of shit in terms of text and training on those tokens is a waste of compute. Just having a good classifier that labelize that is cool. And Llama was at the time, before Llama 3, the best model we had access to legally to labelize the web and select what are the good tokens and the bad tokens. The additional thing is that it also enabled to have a topic tag, like, is it about law? Is it about politics? Is it about chemistry, math, reasoning? So that you can also adapt a bit the mixture to like balance a bit more the diversity.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.