Matthieu Wyart: evaluation

Transcript context

…It's so frustrating that we know so many things that could advance the frontier. But OpenAI and Anthropic, they're still training old school transformers. And spoke to a Cleon Jones at Saqqanra about this. Was 1 of the inventors of the transformer. He said, any new method has to be crushingly better because we've invested so much time in hardware and optimizers and compilers. There's an entire ecosystem about this and it's actually very difficult to steer the ship. Lecun does have a couple of new startups but the read I'm getting is that it's he's focusing on vertical domains. We haven't yet done the moonshot where we try these these new models on mass. Yes. So I agree. I will still say a word of caution that those LLMs are generative models that's very important for them to be because you can interact with them and they produce reasoning and so on. And I still don't know even theoretically if even though I understood all this hierarchy hidden in the world, I can use it efficiently to bring to go back to a prediction at a token level or not. If we can, then if we could, then even it would mean at least conceptually that we could do a much better generative model. But you see, so it's subtle, but there is a distinction between understanding the structure of the world, which is like building an encoder and then decoding it for a very low level aspect of the data. You mentioned diffusion models and you had a great paper out about that. But just conceptually, how do you think they are different from something like a transformer?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.