Andrej Karpathy: recommendation

Transcript context

…Can I ask some questions about teaching well? If you had to give advice to another educator in another field that you’re curious about to make the kinds of YouTube tutorials you’ve made. Maybe it might be especially interesting to talk about domains where you can’t test someone’s technical understanding by having them code something up or something. What advice would you give them? That’s a pretty broad topic. There are 10–20 tips and tricks that I semi-consciously do probably. But a lot of this comes from my physics background. I really, really did enjoy my physics background. I have a whole rant on how everyone should learn physics in early school education because early school education is not about accumulating knowledge or memory for tasks later in the industry. It’s about booting up a brain. Physics uniquely boots up the brain the best because some of the things that they get you to do in your brain during physics is extremely valuable later. The idea of building models and abstractions and understanding that there’s a first-order approximation that describes most of the system, but then there’re second-order, third-order, fourth-order terms that may or may not be present. The idea that you’re observing a very noisy system, but there are these fundamental frequencies that you can abstract away. When a physicist walks into the class and they say, “Assume there’s a spherical cow,” everyone laughs at that, but this is brilliant. It’s brilliant thinking that’s very generalizable across the industry because a cow can be approximated as a sphere in a bunch of ways. There’s a really good book, for example, Scale. It’s from a physicist talking about biology. Maybe this is also a book I would recommend reading. You can get a lot of really interesting approximations and chart scaling laws of animals. You can look at their heartbeats and things like that, and they line up with the size of the animal and things like that. You can talk about an animal as a volume. You can talk about the heat dissipation of that, because your heat dissipation grows as the surface area, which is growing as a square. But your heat creation or generation is growing as a cube. So I just feel like physicists have all the right cognitive tools to approach problem solving in the world. So because of that training, I always try to find the first-order terms or the second-order terms of everything. When I’m observing a system or a thing, I have a tangle of a web of ideas or knowledge in my mind. I’m trying to find, what is the thing that matters? What is the first-order component? How can I simplify it? How can I have a simplest thing that shows that thing, shows it in action, and then I can tack on the other terms? Maybe an example from one of my repos that I think illustrates it well is called micrograd. I don’t know if you’re familiar with this. So micrograd is 100 lines of code that shows backpropagation. You can create neural networks out of simple operations like plus and times, et cetera. Lego blocks of neural networks. You build up a computational graph and you do a forward pass and a backward pass to get the gradients. Now, this is at the heart of all neural network learning. So micrograd is a 100 lines of pretty interpretable Python code, and it can do forward and backward arbitrary neural networks, but not efficiently. this is at the heart of all neural network learning. So micrograd is a 100 lines of pretty interpretable Python code, and it can do forward and backward arbitrary neural networks, but not efficiently. So micrograd, these 100 lines of Python, are everything you need to understand how neural networks train. Everything else is just efficiency. Everything else is efficiency. There’s a huge amount of work to get efficiency. You need your tensors, you lay them out, you stride them, you make sure your kernels, orchestrating memory movement correctly, et cetera. It’s all just efficiency, roughly speaking. But the core intellectual piece of neural network training is micrograd. It’s 100 lines. You can easily understand it. It’s a recursive application of chain rule to derive the gradient, which allows you to optimize any arbitrary differentiable function. So I love finding these small-order terms and serving them on a platter and discovering them. I feel like education is the most intellectually interesting thing because you have a tangle of understanding and you’re trying to lay it out in a way that creates a ramp where everything only depends on the thing before it. I find that this untangling of knowledge is just so intellectually interesting as a cognitive task. I love doing it personally, but I just have a fascination with trying to lay things out in a certain way. Maybe that helps me.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.