Ramin Hasani: belief

Transcript context

…You know, when you think about intelligence, like the way I look at it right now is that like, Transformer-based networks, and also our type of, let's say, the current architectures, the landscape of architectures that are available, they gave us with scale, they gave us in-context learning capability. The thing that actually emerged from next token, you see, the word is important, emerged from next token prediction. Intelligence for me is an emergent property. If you want to get into, if you want to miniaturize kind of intelligence, bring it to the physical world, I don't believe with the current set of algorithms, you would be able to get close to the, let's say, intelligence per watt that human brain is actually providing, right? You're not gonna get there. Why? Because I believe... human brain over the... We also have to consider the amount of energy that went into design of humans as a whole of biological evolution. It's actually a very, very long kind of process. And a lot of the pre-training process that people say, Oh, a lot of... You're going to see the entire world. Humans do not need to see the entire internet to be able to reason about something. No, but humans have gone through... years of evolution. So I would actually attribute it a lot to the evolutionary kind of aspect of things. But again, what I would say, human intelligence came with multiple mechanisms of in-context learning. You just don't have, when our current AI systems do in-context learning, they learned a vague representation of one algorithm, which is least square. It's basically least the square. So what they figured out is basically gradient descent in a mushy way, like for some use cases, With some examples, you would be able to make the system understand and give you kind of the next example. These are kind of beautiful properties. This is what I would call an emergent property of the systems. You set the algorithm to be next token prediction. You got a vague version of a gradient descent in context learning capabilities. For humans, You don't have only emergent graduates. You can learn by examples, but you can also do reinforcement learning. You can run simulations in your head. You can do all sort of algorithms. You can do Bayesian statistics in your brain. You see what I'm saying? So you have like a diverse set of algorithms emerged from the way that humans actually got designed and got intelligence. For me, I believe if you start optimizing for those, if you try to force your way into the system to become a reinforcement learning, not learner, but learning from trajectories, that's not the right way to actually get to emergent property of intelligence. ou try to force your way into the system to become a reinforcement learning, not learner, but learning from trajectories, that's not the right way to actually get to emergent property of intelligence. I believe the format of intelligence that is going to be miniaturized, you know, like being like in a smallest amount of time, maximum amount of intelligence, like the human brain, we need to come up with these identifiers, like these confounding kind of algorithms, which is next token prediction is one of those. What else do we have to do at the beginning of the design of these systems so that reinforcement learning, like curiosity-driven like intelligence, drive with limited amount of energy from that final system that actually comes out of it? So I would say that emergent property is something that we need, there's a lot of research that has to go in that direction, I would say. And multiple kind of variations of way of learning, that would enable the next generation of artificial intelligence, I think. That could be a great note to end on, but let me just give you one more opportunity. Anything else that I didn't touch on or anything else you just would want to leave people with before we break?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.