Shawn Wang: commitment

Transcript context

…Yeah, I think it would be amazing if we could replicate the same on the open models to then, because now we can use Llama 3.1 to generate synthetic data for training and fine-tuning. I think, obviously, Anthropic has a lot of compute and a lot of money. So once they figure out, OK, this is what we should make the model better at, they can put a lot of resources. I think an open source is probably going to be a more distributed effort. I feel like Noose has held the crown of the best fine-tuning data site owners for a while, but at some point that should change, hopefully. Other groups should step up. And I think if we can apply the same principles to a model as big as 405B and bring them into maybe the 7B form factor, that would be great. But yeah, Cloud is great. I canceled JGBD a while ago. Really small podcaster run for latent space. It runs both on Cloud and on OpenAI, and Cloud is definitely better most of the time. It's not a benchmark. It's just vibes. But when the vibes are good, the vibes are good. We run most of the AI news summaries on Cloud as well. And I always run it against OpenAI. Sometimes OpenAI wins. I do a daily comparison. But yeah, Cloud is very strong at summarization and instruction following, which is something I care a lot about. So when you talk about frontier models, MMLU no longer cut it. We have reached 92 on MMLU. It's going to 95, 97. It just means you're memorizing MMLU. There's some fundamental irreducible level of mistakes because of MMLU's quality. We talked about this with Clementine on the Hugging Face episode. And so we need to see what else. What is the next frontier? I think there are 10 directions that I outlined below, but we'll talk about that later. Yeah. Should we move on to number three? Yeah. 3.1. I guess that to make sure to differentiate between the models.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.