Andrej Karpathy: uncertainty

Transcript context

…Right, but why is the distillation in 10 years not getting below 1 billion? Oh, you think it should be smaller than a billion? I mean, come on, right? I don’t know. At some point it should take at least a billion knobs to do something interesting. You’re thinking it should be even smaller? Yeah. If you look at the trend over the last few years of just finding low-hanging fruit and going from trillion plus models to models that are literally two orders of magnitude smaller in a matter of two years and having better performance, it makes me think the sort of core of intelligence might be even way, way smaller. Plenty of room at the bottom, to paraphrase Feynman.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.