Andrej Karpathy does not fully resonate with the idea that continual learning over longer horizons will emerge spontaneously in large language models as in-context learning did.
I don't know that I fully resonate with that. These models, when you boot them up and they have zero tokens in the window, they're always restarting from scratch where they were.