Andrej Karpathy: belief

Transcript context

…Just to steelman the other perspective, after doing this Sutton interview and thinking about it a bit, he has an important point here. Evolution does not give us the knowledge, really. It gives us the algorithm to find the knowledge, and that seems different from pre-training. Perhaps the perspective is that pre-training helps build the kind of entity which can learn better. It teaches meta-learning, and therefore it is similar to finding an algorithm. But if it’s “Evolution gives us knowledge, pre-training gives us knowledge,” that analogy seems to break down. It’s subtle and I think you’re right to push back on it, but basically the thing that pre-training is doing, you’re getting the next-token predictor over the internet, and you’re training that into a neural net. It’s doing two things that are unrelated. Number one, it’s picking up all this knowledge, as I call it. Number two, it’s actually becoming intelligent. By observing the algorithmic patterns in the internet, it boots up all these little circuits and algorithms inside the neural net to do things like in-context learning and all this stuff. You don’t need or want the knowledge. I think that’s probably holding back the neural networks overall because it’s getting them to rely on the knowledge a little too much sometimes. For example, I feel agents, one thing they’re not very good at, is going off the data manifold of what exists on the internet. If they had less knowledge or less memory, maybe they would be better. What I think we have to do going forward—and this would be part of the research paradigms—is figure out ways to remove some of the knowledge and to keep what I call this cognitive core. It’s this intelligent entity that is stripped from knowledge but contains the algorithms and contains the magic of intelligence and problem-solving and the strategies of it and all this stuff. There’s so much interesting stuff there. Let’s start with in-context learning. This is an obvious point, but I think it’s worth just saying it explicitly and meditating on it. The situation in which these models seem the most intelligent—in which I talk to them and I’m like, “Wow, there’s really something on the other end that’s responding to me thinking about things—is if it makes a mistake it’s like, “Oh wait, that’s the wrong way to think about it. I’m backing up.” All that is happening in context. That’s where I feel like the real intelligence is that you can visibly see. That in-context learning process is developed by gradient descent on pre-training. It spontaneously meta-learns in-context learning, but the in-context learning itself is not gradient descent, in the same way that our lifetime intelligence as humans to be able to do things is conditioned by evolution but our learning during our lifetime is happening through some other process.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.