Shawn Wang: belief

Transcript context

…Yeah, it almost feels like you, you definitely cannot trust that. And for computer vision, that's why I like spend a lot of time with the customers being like, is this a valid set of tests? Like, is this truly your test environment? Is this exactly what you need to be validating against? And how do we get to that point where you have something that you can validate against was quite, quite challenging. But that was, uh, the bigger. We had there, I would say to bring people up to speed as well in like very recent developments. Have you come across fine web? It's a data set from Hugging Face that is kind of like a cleaned C4 and they use LLMs to not to distill, but to actually filter. And to improve data quality using LLMs to filter that model seems to be unexplored. And the initial results from the LLM. c project is that you can train the same quality of model for like basically 10x less tokens. So, trading with 10 billion tokens versus 100 billion tokens on the GPT 2 architecture seems to get you the same, or even slightly better, perplexity and eval scores, which is interesting that it's not quite synthetic data, but it's also just data quality improvement in other formats. Exactly. With synthetic data, we saw that if we just got you the right distribution of data that fit what you needed in the real world, then that was it. And you didn't have to train with as many samples as you needed otherwise. In a way, I see it like training. a, child in like Exeter, right? It doesn't matter how smart the child is because the information is being fed to it so well, in particular, like, you know, there are really incredible schools that fit the information to you really well and the right information. And by doing that as a human that works, I don't see why that doesn't work. It doesn't work with this kind of models and we saw it working in computer vision. It was just very small data set, just the right data, fit it well, and it will work. Um, yeah. And that was the experience.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.