Andrej Karpathy: evaluation

Transcript context

…Stepping back, what is the part about human intelligence that we have most failed to replicate with these models? Just a lot of it. So maybe one way to think about it, I don’t know if this is the best way, but I almost feel like — again, making these analogies imperfect as they are — we’ve stumbled by with the transformer neural network, which is extremely powerful, very general. You can train transformers on audio, or video, or text, or whatever you want, and it just learns patterns and they’re very powerful, and it works really well. That to me almost indicates that this is some piece of cortical tissue. It’s something like that, because the cortex is famously very plastic as well. You can rewire parts of brains. There were the slightly gruesome experiments with rewiring the visual cortex to the auditory cortex, and this animal learned fine, et cetera. So I think that this is cortical tissue. I think when we’re doing reasoning and planning inside the neural networks, doing reasoning traces for thinking models, that’s kind of like the prefrontal cortex. Maybe those are like little checkmarks, but I still think there are many brain parts and nuclei that are not explored. For example, there’s a basal ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning. But where’s the hippocampus? Not obvious what that would be. Some parts are probably not important. Maybe the cerebellum is not important to cognition, its thoughts, so maybe we can skip some of it. But I still think there’s, for example, the amygdala, all the emotions and instincts. There’s probably a bunch of other nuclei in the brain that are very ancient that I don’t think we’ve really replicated. I don’t know that we should be pursuing the building of an analog of a human brain. I’m an engineer mostly at heart. Maybe another way to answer the question is that you’re not going to hire this thing as an intern. It’s missing a lot of it because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models. So it’s not fully there yet. You can look at it as not all the brain parts are checked off yet. This is maybe relevant to the question of thinking about how fast these issues will be solved. Sometimes people will say about continual learning, “Look, you could easily replicate this capability. Just as in-context learning emerged spontaneously as a result of pre-training, continual learning over longer horizons will emerge spontaneously if the model is incentivized to recollect information over longer horizons, or horizons longer than one session.” So if there’s some outer loop RL which has many sessions within that outer loop, then this continual learning where it fine-tunes itself, or it writes to an external memory or something, will just emerge spontaneously. Do you think things like that are plausible? I just don’t have a prior over how plausible that is. How likely is that to happen?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.