Shawn Wang: belief

Transcript context

…I think to me, the biggest thing is the training on outputs. Every company is just hiding the fact that they've been fine tuning and training on GPT-4 outputs. And you can not technically do it, but obviously OpenAI is not enforcing it. I think now for the first time, there's a clear path to how do we make a 7b model good without having to go through GPT-4 or going to Cloud 3. And we'll kind of talk about this later, but I think we're seeing maybe the, not the death, but settling the picks and shovels, it's kind of going away. And building the vertical things is where most of the value is actually getting captured, at least at the early stages. So being able to make small models better at specific things through a large model, it's more important than yet another 7b model that I can try and use. But at the end of the day, I still need to go through the large labs to fine tune. So that to me is the most interesting thing. It's such a large model. It's obviously amazing, but I don't know if a lot of people are switching from GPT-4 or Cloud 3.5 to run 4 or 5b. I also don't know what the hosting options are as far as scaling. I don't know if the fireworks and togethers of the world, how much capacity they actually have to serve this model. Because at the end of the day, it's a lot of compute if some of the big products will switch to it and you cannot easily run it yourself. So I don't know. But to me, the synthetic data piece is definitely the most interesting. Yeah. I would say that it is not enough now to say that synthetic data is real. I actually shipped that in the original email and then I changed that in the sort of what you see now in the podcast description. But because it is so established now that synthetic data is real, therefore you need to go to the next level, which is, OK, what do you use it for and how do you use it? And I think that is what was interesting for Lama3 for me. If you read the paper, 90 pages of all filler no killer is something like that. This is what the people were saying. Very, very for once a frontier model with proper paper instead of a marketing blog post. And, you know, they actually spelled out how they do synthetic data for a few different domains. So they have synthetic data for code, for math, for multilinguality, for long context, for tool use, and then also for ASR and voice generation. And I think that, OK, now you have the license to go distill Lama3, Lama4, Lama5B. But how do you do that? That is the sort of the next frontier. Now you have the permission to do it. How do you do it? And I think that people are going to reference Lama3 a lot, but then they can use those techniques for everything else. You know, in our episode with Thomas, he talked about, like, I was very focused on synthetic data for pre-training because that's my context. That's my conversations with Technium from Noose and all the other people doing synthetic data for pre-training and fine tuning. But he was talking about post-training as well. And for everything here was post-training. In fact, I wish we had spent more time with Thomas on this stuff. We just didn't have the paper beforehand. But I think, like, when I call Lama3, the synthetic data model is you have the license for it, but then you also have the roadmap, the recipe, because it's in the paper. And now, like, now everybody knows how to do this. And probably, you know, obviously, like, opening eyes probably laughing at us because they did this a year ago. But now it's in the open. I mean, they can laugh all they want, but they're coming for them. I think, I mean, that's definitely the biggest vibe shift, right? It's like, obviously Lama3.1 is good. Obviously, Claude is good. Maybe a year and a half ago, you didn't get the benefit of the doubt. It's like an open AI competitor to be state of the art. You know, it was kind of like, oh, Entropic, yeah, those guys are cute over there. They're trying to do their thing, but it's not open AI. And like, Lama2 is great, but like, it's really not a serious model. You know, it's like just good enough. I think now it's like every time Entropic releases something, people are like, okay, this is like a serious thing. Whenever like Meta releases something, it's like, okay, they're at the same level. And I don't know if open AI is kind of like sandbagging the GBT next.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.