Alessio Fanelli: belief

Transcript context

…I'm not sure if it's this one, maybe. But basically, the idea was that with a mixture of experts, you have an expert that is an identity matrix that you can skip. And so you can... But that's early works, very preliminary works. For instance, I haven't seen yet a lot like putting the compute, generating a token into the loss. That's going to be interesting when we start to do that. I know we're getting up on time, but we have just a few more questions we definitely want to ask you. So as you think about... There were reports about Llama4 started training again in June. If you think about the evolution of the models, I think up until Llama3, with Meta AI and some of these things, I'm like, it makes sense that they want to build their own models and they're multi-modal. It sounds like Llama4, maybe a lot of the focus will also be a more agentic behavior and have all of this. I'm curious at what point it's like, okay, this is a research direction that we still want to take, even though it doesn't fit right into the product. What's that discussion internally about what to focus on as you keep scaling these models? Yeah. I think it's a balance between, well, we want to be number one, Mark wants to be number one there. And there's this understanding also that this is a critical technology in the future. And even if nowadays that research, if nowadays it's not directly intersecting product, we don't want to be late in the game as we had in the past. So that's the first thing. The second thing is, we think that this technology will change the world. We want to work towards AGI and AGI will change the world. And if Meta develop an AGI, it will probably intersect pretty easily the products. Now the third thing is, with that in mind, we have to balance with product needs. And there's always this ongoing discussion and this balance to find for like between a flagship model, between maybe a model that will be more adapted to product needs. And it doesn't have to be decorrelated. As I said before, like you can leverage also the big models to distillate some capabilities to a smaller one that will be maybe more suited like research. There's always this back and forth. There's also the fact that the product kind of ideas to the research evaluations that are grounded in actual use cases, that we can also measure ourselves with respect to is there some progress or is it just on an academic benchmark, you know?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.