Ramin Hasani: belief

Transcript context

…t frame of mind to think about that kind of feature. And for us, like we think that this input dependence is one of the fundamental discoveries that we have done, like that we learned also in biology. It happens, you know, like it comes out of physics. Like if you just put the math together, you're going to realize like, oh, what is special about this dynamical system is the fact that it has this non-linear input dependence. That's kind of a structure. And by biases, I mean the complexity of that gating. Even existence of that gating is a bias that you're adding to your system. Now, is it needed or not at scale? That's a big question mark. You're going to see when neural networks at 100 trillion parameters, is matrix multiplication is enough to really perform general purpose computer, get to AGIs of the world or not? big question mark. You're going to see when neural networks at 100 trillion parameters, is matrix multiplication is enough to really perform general purpose computer, get to AGIs of the world or not? That's a big question mark. You need to also put this discussion about architecture and the obsession about architecture in perspective. Another perspective you have to put this in is the learning theories themselves. Right now, we're talking about different learning schemes that are coming. For example, you can train a model with next token prediction. You can train them in a word modeling kind of context. You can train them in a sequential, in a long-term horizon and long-term history. There's so many ways you can construct also the objective function of the learning algorithm itself that would also contributes a lot to the learning process. Architecture, I'll tell you that we found the purpose of the architecture and the most important application of architectural kind of research has been inefficiency, right? Really getting to efficient formats of competition without lots of quality. extremely important thing because we're talking about the resource allocation problem right now. As the models are becoming bigger and the demand of AI is exponentially going higher, you want to have more efficient versions of the systems to actually run the systems at scale. Otherwise, how can we actually provide access to AI to all? And we are seeing early instances of this thing that the larger labs are wiping out the compute off of the planet. Why? Because they have to train and they have to then host these models for all of us, right? So efficiency becomes a fundamental property of this model architectures that we are doing the research on. I would say if you want to get into architectures that enable next generation of neural networks, sorry, next generation of intelligent systems in a way that they can become like human brain that actually performs, you know, like performs computation with 20 watts of power and it's already an AGI that does that between, that's like the massive Exploration that goes beyond architecture, it goes into memory research, it goes into learning algorithms kind of feature, it goes into data feature, you know, it goes into prior research, you know, like looking at it from a statistical learning perspective. That becomes kind of a... And then learning theory itself, like the limitations of learning theory itself, it also gets imposed on today's learning systems because the definition of learning theory itself is actually broke at scale. But what we have right now, everything is IID. We are building averaging machines. You've seen the quality of writing and the sycophancy that actually comes out of AI systems because of the fact that we haven't gotten... Maybe to some extent, multi-agents has been becoming the solution and test time kind of scale. became kind of a solution for the caveats that you're seeing from just learning systems with autoregressive kind of modeling as a pre-training kind of method. But I think there will be innovations needed, not just on architecture, but the whole thing as a whole. rom just learning systems with autoregressive kind of modeling as a pre-training kind of method. But I think there will be innovations needed, not just on architecture, but the whole thing as a whole. You have data, you have algorithms, you have data, you have models, and you have learning algorithms. All together, they can design basically a future, that ultimate kind of holy grail, basically, in this space.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.