Andrej Karpathy: prediction

Transcript context

…Do you expect the low-hanging fruit to be similar in kind to the kinds of things that have been happening over the last two to five years? If I look at nanochat versus nanoGPT and the architectural tweaks you made, is that the flavor of things you expect to continue to keep happening? You’re not expecting any giant paradigm shifts. For the most part, yeah. I expect the datasets to get much, much better. When you look at the average datasets, they’re extremely terrible. They’re so bad that I don’t even know how anything works. Look at the average example in the training set: factual mistakes, errors, nonsensical things. Somehow when you do it at scale, the noise washes away and you’re left with some of the signal. Datasets will improve a ton. Everything gets better. Our hardware, all the kernels for running the hardware and maximizing what you get with the hardware. Nvidia is slowly tuning the hardware itself, Tensor Cores, all that needs to happen and will continue to happen. All the kernels will get better and utilize the chip to the max extent. All the algorithms will probably improve over optimization, architecture, and all the modeling components of how everything is done and what the algorithms are that we’re even training with. I do expect that nothing dominates. Everything plus 20%. This is roughly what I’ve seen. People have proposed different ways of charting how much progress we’ve made towards full AGI. If you can come up with some line, then you can see where that line intersects with AGI and where that would happen on the x-axis. People have proposed it’s the education level. We had a high schooler, and then they went to college with RL, and they’re going to get a Ph.D.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.