Shawn Wang: prediction

Transcript context

…Right. So it's very interesting, the research direction. Synthetic data in general, synthetic data for pre-training. My intuition is that the web is full of shit in terms of text and training on those tokens is a waste of compute. Just having a good classifier that labelize that is cool. And Llama was at the time, before Llama 3, the best model we had access to legally to labelize the web and select what are the good tokens and the bad tokens. The additional thing is that it also enabled to have a topic tag, like, is it about law? Is it about politics? Is it about chemistry, math, reasoning? So that you can also adapt a bit the mixture to like balance a bit more the diversity. To me, you know, I'm not exactly sure what you guys did, but like, I feel like when people say synthetic data, there needs to be different categories of synthetic data now, because I think there's so many different usage of this thing. But specifically synthetic data for pre-training, it feels almost like you're running multiple epochs on the raw data while it's rephrased or reformatted by a language model, right? And in my mind, it's very similar to computer vision, where you do data augmentation on an item, right? Like we're doing data augmentation. That's the less cool name for synthetic data. That's very interesting. I totally agree with you related to pre-training, totally stamp what you said. I think it's very different though for post-training and the future direction on synthetic data that I'm personally excited. Like for instance, what I'm excited about is we had this survey on augmented LLM a year ago. And all the idea is like, if you augment your LLM with something else, it can be a retriever. It can be search. It can be a tool. It can be a calculator. It can be a code execution. Then you are not just doing some data augmentation with your model, but you're actually adding some expert skills that possibly goes beyond the model weights. For instance, if your model can calculate something it was wrong before and now it has access to a calculator and you can retrain your model on that, then you're learning something new. If your model didn't know something about LLM 2, probably doesn't know a lot about LLM 3. You can search online about it and then you train the model on that. Then you have a positive feedback loop, like what we call expert direction, targeting directly the weakness of the model. It's like continual augmentation of the language model, much beyond just data augmentation.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.