Andrej Karpathy

I will say that there have been some papers that I thought were interesting that look at the mechanisms behind in-context learning.

But we're going to converge on a similar architecture cognitively. In 10 years, do you think it'll still be something like a transformer, but with much more modified attention and more sparse MLPs and so forth?

If I can't build it, I don't understand it. That’s a Feynman quote, I believe. I 100% have always believed this very strongly, because there are all these micro things that are just not properly arranged and you don't really have the knowledge.

What is the best way for somebody to learn from it? Is it to just delete all the code and try to reimplement from scratch, try to add modifications to it? That's a great question. Basically it's about 8,000 lines of code that takes you through the entire pipeline. I would probably put it on the right monitor. If you have two monitors, you put it on the right. You want to build it from scratch, you build it from the start. You're not allowed to copy-paste, you're allowed to reference, you're not allowed to copy-paste.

So that part is not there yet. I would love to add that probably later this week. It's probably a video or something like that.

The other part is when I was rewriting the tokenizer in Rust. I'm not as good at Rust because I'm fairly new to Rust. So there's a bit of vibe coding going on when I was writing some of the Rust code. But I had a Python implementation that I fully understand, and I'm just making sure I'm making a more efficient version of it, and I have tests so I feel safer doing that stuff.

it makes me think the sort of core of intelligence might be even way, way smaller. Plenty of room at the bottom, to paraphrase Feynman.

Our hardware, all the kernels for running the hardware and maximizing what you get with the hardware. Nvidia is slowly tuning the hardware itself, Tensor Cores, all that needs to happen and will continue to happen.

Today, what jobs are replaceable by AI? A good example recently was Geoff Hinton's prediction that radiologists would not be a job anymore, and this turned out to be very wrong in a bunch of ways.

Self-driving is very interesting because it's definitely where I get a lot of my intuitions because I spent five years on it. It has this entire history where the first demos of self-driving go all the way to the 1980s. You can see a demo from CMU in 1986.

For example, Claude Code or OpenAI Codex and stuff like that didn't even exist a year ago. Is that right? This is a miraculous technology that didn't exist. There's going to be a huge amount of demand, as we see the demand in ChatGPT already and so on.

For example, Claude Code or OpenAI Codex and stuff like that didn't even exist a year ago. Is that right? This is a miraculous technology that didn't exist. There's going to be a huge amount of demand, as we see the demand in ChatGPT already and so on.

For example, Claude Code or OpenAI Codex and stuff like that didn't even exist a year ago. Is that right? This is a miraculous technology that didn't exist. There's going to be a huge amount of demand, as we see the demand in ChatGPT already and so on.

Even today, I would say ChatGPT is an extremely valuable educational product.

I'm building the first course. I want to have a really, really good course, the obvious state-of-the-art destination you go to to learn, AI in this case.

I'm building the first course. I want to have a really, really good course, the obvious state-of-the-art destination you go to to learn, AI in this case.

With what’s happened so far with online courses, why haven't they already enabled us to enable every single human to know everything? They're just so motivation-laden because there are no obvious on-ramps and it's so easy to get stuck.

What I did was I used ChatGPT to ask the questions with the paper in the context window. It worked through some of the simple things. Then I shared the thread to the person who wrote that paper or worked on that work. I felt like if they could see the dumb questions I had, it might help them explain better in the future.

I was triggered by that because there's some over-prediction going on in the industry. In my mind, this is more accurately described as the decade of agents.

They don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this stuff. They don't do a lot of the things you've alluded to earlier. They don't have continual learning.

It will take about a decade to work through all of those issues.

If somebody asks how long continual learning will take, I have no prior about whether this is a project that should take 5 years, 10 years, or 50 years.

I've been in AI for almost two decades. It's going to be 15 years or so, not that long.

I feel like the problems are tractable, they're surmountable, but they're still difficult. If I just average it out, it just feels like a decade to me.

I still think there will continue to be some because they come with almost surprising regularity.

the weights of the neural net are not stored in ATCGs. There's some dramatic compression.

Evolution seems closer to finding the algorithm which then does the lifetime learning.

That's why I call pre-training this crappy evolution. It's the practically possible version with our technology and what we have available to us to get to a starting point where we can do things like reinforcement learning and so on.

It's doing two things that are unrelated. Number one, it's picking up all this knowledge, as I call it. Number two, it's actually becoming intelligent. By observing the algorithmic patterns in the internet, it boots up all these little circuits and algorithms inside the neural net to do things like in-context learning and all this stuff.

You don't need or want the knowledge. I think that's probably holding back the neural networks overall because it's getting them to rely on the knowledge a little too much sometimes.

What I think we have to do going forward—and this would be part of the research paradigms—is figure out ways to remove some of the knowledge and to keep what I call this cognitive core.

That's where I feel like the real intelligence is that you can visibly see. That in-context learning process is developed by gradient descent on pre-training.

In-context learning is pattern completion within a token window.

I will say that there have been some papers that I thought were interesting that look at the mechanisms behind in-context learning.

if you give it the full chapter and ask it questions, you're going to get much better results because it's now loaded in the working memory of the model.

we've stumbled by with the transformer neural network, which is extremely powerful, very general. You can train transformers on audio, or video, or text, or whatever you want, and it just learns patterns and they're very powerful, and it works really well.

the cortex is famously very plastic as well. You can rewire parts of brains. There were the slightly gruesome experiments with rewiring the visual cortex to the auditory cortex, and this animal learned fine, et cetera.

when we're doing reasoning and planning inside the neural networks, doing reasoning traces for thinking models, that's kind of like the prefrontal cortex.

there's a basal ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning.

where's the hippocampus? Not obvious what that would be. Some parts are probably not important. Maybe the cerebellum is not important to cognition, its thoughts, so maybe we can skip some of it. But I still think there's, for example, the amygdala, all the emotions and instincts.

I don't know that we should be pursuing the building of an analog of a human brain. I'm an engineer mostly at heart.

you're not going to hire this thing as an intern. It's missing a lot of it because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models. So it's not fully there yet.

I don't know that I fully resonate with that. These models, when you boot them up and they have zero tokens in the window, they're always restarting from scratch where they were.

We don't have an equivalent of that in large language models. That's to me more adjacent to when you talk about continual learning and so on as absent.

But we're going to converge on a similar architecture cognitively. In 10 years, do you think it'll still be something like a transformer, but with much more modified attention and more sparse MLPs and so forth?

But I still think that humans obviously have some process for distilling some of that knowledge into the weights. We're missing it.

Build the stuff yourself, but don't allow yourself copy-paste. I do think that there's two types of knowledge, almost. There's the high-level surface knowledge, but when you build something from scratch, you're forced to come to terms with what you don't understand and you don't know that you don't understand it. It always leads to a deeper understanding.

If I can't build it, I don't understand it. That’s a Feynman quote, I believe. I 100% have always believed this very strongly, because there are all these micro things that are just not properly arranged and you don't really have the knowledge.

What is the best way for somebody to learn from it? Is it to just delete all the code and try to reimplement from scratch, try to add modifications to it? That's a great question. Basically it's about 8,000 lines of code that takes you through the entire pipeline. I would probably put it on the right monitor. If you have two monitors, you put it on the right. You want to build it from scratch, you build it from the start. You're not allowed to copy-paste, you're allowed to reference, you're not allowed to copy-paste.

When you write this code, you don't go from top to bottom, you go from chunks and you grow the chunks, and that information is absent. You wouldn't know where to start. So it's not just a final repository that's needed, it's the building of the repository, which is a complicated chunk-growing process.

So that part is not there yet. I would love to add that probably later this week. It's probably a video or something like that.

You tweeted out that coding models were of very little help to you in assembling this repository. I'm curious why that was.

hey keep trying to make a production code base, and I have a bunch of assumptions in my code, and it's okay. I don't need all this extra stuff in there. So I feel like they're bloating the code base, bloating the complexity

they keep misunderstanding, they're using deprecated APIs a bunch of times. It's a total mess. It's just not net useful.

I also feel like it's annoying to have to type out what I want in English because it's too much typing. If I just navigate to the part of the code that I want, and I go where I know the code has to appear and I start typing out the first few letters, autocomplete gets it and just gives you the code.

The other part is when I was rewriting the tokenizer in Rust. I'm not as good at Rust because I'm fairly new to Rust. So there's a bit of vibe coding going on when I was writing some of the Rust code. But I had a Python implementation that I fully understand, and I'm just making sure I'm making a more efficient version of it, and I have tests so I feel safer doing that stuff.

They're not very good at code that has never been written before, maybe it's one way to put it, which is what we're trying to achieve when we're building these models.

Reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible.

In reinforcement learning, say you're solving a math problem, because it's very simple. You're given a math problem and you're trying to find the solution. In reinforcement learning, you will try lots of things in parallel first.

The way I like to put it is you're sucking supervision through a straw.

To the extent you think this is the bottleneck to making RL more functional, then that will require making LLMs better judges, if you want to do this in an automated way.

There's an infinity of adversarial examples.

I am only at a stage of reading abstracts because a lot of these papers are just ideas. Someone has to make it work on a frontier LLM lab scale in full generality because when you see these papers, they pop up, and it's just a bit noisy. They're cool ideas, but I haven't seen anyone convincingly show that this is possible.

The book is a set of prompts for me to do synthetic data generation,

When you're reading a book, I don't even feel like the book is exposition I'm supposed to be attending to and training on. The book is a set of prompts for me to do synthetic data generation, or for you to get to a book club and talk about it with your friends.

Humans are a lot noisier, but at least they're not biased, in a statistical sense. They're not silently collapsed. They maintain a huge amount of entropy.

Humans collapse during the course of their lives. This is why children, they haven't overfit yet.

This is why children, they haven't overfit yet. They will say stuff that will shock you because you can see where they're coming from, but it's just not the thing people say, because they're not yet collapsed.

when you're generating things in your head and then you're attending to it, you're training on your own samples, you're training on your synthetic data. If you do it for too long, you go off-rails and you collapse way too much.

The best learners that we are aware of, which are children, are extremely bad at recollecting information. In fact, at the very earliest stages of childhood, you will forget everything. You're just an amnesiac about everything that happens before a certain year date. But you're extremely good at picking up new languages and learning from the world.

if you compare it to the opposite end of the spectrum, you have LLM pre-training, where these models will literally be able to regurgitate word-for-word what is the next thing in a Wikipedia page. But their ability to learn abstract concepts really quickly, the way a child can, is much more limited.

I do think that humans have a lot more of an element, compared to LLMs, of seeing the forest for the trees. We're not actually that good at memorization, which is actually a feature. Because we're not that good at memorization, we're forced to find patterns in a more general sense.

LLMs in comparison are extremely good at memorization. They will recite passages from all these training sources. You can give them completely nonsensical data. You can hash some amount of text or something like that, you get a completely random sequence. If you train on it, even just for a single iteration or two, it can suddenly regurgitate

Whereas LLMs are distracted by all the memory that they have of the pre-training documents, and it's probably very distracting to them in a certain sense.

I'd love to have them have less memory so that they have to look things up, and they only maintain the algorithms for thought, and the idea of an experiment, and all this cognitive glue of acting.

The models are way too good at memorization, and somehow we should remove that.

I feel like the diversity of the outputs is not so much... Number one, it's much harder to work with and evaluate and all this stuff, but maybe it's not what's capturing most of the value.

If you're super creative in RL, it's not good. Yeah. Or maybe if you're doing a lot of writing, help from LLMs and stuff like that, it's probably bad because the models will silently give you all the same stuff.

Possibly the labs should try harder. I think you hinted that it's a very fundamental problem, it won't be easy to solve.

I do think that you could probably regularize the entropy to be higher. So you're encouraging the model to give you more and more solutions, but you don't want it to start deviating too much from the training data.

distillation works extremely well. So almost every small model, if you have a small model, it's almost certainly distilled.

it makes me think the sort of core of intelligence might be even way, way smaller. Plenty of room at the bottom, to paraphrase Feynman.

We had increasing scale up to maybe GPT 4.5 and now we're seeing decreasing or plateauing scale.

The labs are just being practical. They have a flops budget and a cost budget. It just turns out that pre-training is not where you want to put most of your flops or your cost. That's why the models have gotten smaller.

I expect the datasets to get much, much better. When you look at the average datasets, they're extremely terrible.

Our hardware, all the kernels for running the hardware and maximizing what you get with the hardware. Nvidia is slowly tuning the hardware itself, Tensor Cores, all that needs to happen and will continue to happen.

All the algorithms will probably improve over optimization, architecture, and all the modeling components of how everything is done and what the algorithms are that we're even training with.

I do expect that nothing dominates. Everything plus 20%. This is roughly what I've seen.

I was pretty happy with that at the time. I've stuck to that definition forever, and then people have made up all kinds of other definitions. But I like that definition.

I don't know the numbers. I feel about 10% to 20%, if I had to guess, is only knowledge work, someone could work from home and perform tasks, something like that.

What is the size of the economy, and what is 10% or 20%? We're still talking about a few trillion dollars, even in the US, of market share or work.

Today, what jobs are replaceable by AI? A good example recently was Geoff Hinton's prediction that radiologists would not be a job anymore, and this turned out to be very wrong in a bunch of ways.

Call center employees have a number of simplifying properties with respect to what's automatable today. Their jobs are pretty simple. It's a sequence of tasks, and every task looks similar.

I expect that we are not going to instantly replace people. We're going to be swapping in AIs that do 80% of the volume. They delegate 20% of the volume to humans, and humans are supervising teams of five AIs doing the call center work that's more rote.

I would be looking for new interfaces or new companies that provide some layer that allows you to manage some of these AIs that are not yet perfect.

I do think that code is pretty structured. Text is maybe a lot more flowery, and there's a lot more entropy in text, I would say.

Also code is hard, and so people feel quite empowered by LLMs, even from simple knowledge.

I see it as a progression of automation in society. Extrapolating the trend of computing, there will be a gradual automation of a lot of things, and superintelligence will an extrapolation of that.

We expect more and more autonomous entities over time that are doing a lot of the digital work and then eventually even the physical work some amount of time later.

I think it will. It is fundamentally automation, but it will be extremely foreign. It will look really strange.

I think that's the most likely outcome, that there will be a gradual loss of understanding. We'll gradually layer all this stuff everywhere, and there will be fewer and fewer people who understand it. Then there will be a gradual loss of control and understanding of what's happening.

It is not the fact that they are smarter than us that is resulting in the loss of control. It's the fact that they are competing with each other, and whatever arises out of that competition leads to the loss of control.

We were talking about how currently it feels like when you're doing AI engineering or AI research, these models are more in the category of compiler rather than in the category of a replacement.

You think it's continuous with this hyper-exponential trend? Yes. That's why this was very interesting to me, because I was trying to find AI in the GDP for a while.

You can't find them in GDP. GDP is the same exponential. Even the early iPhone didn't have the App Store, and it didn't have a lot of the bells and whistles that the modern iPhone has.

Everything is so spread out and it so slowly diffuses that everything ends up being averaged up into the same exponential.

With AI we're going to see the exact same thing. It's just more automation. It allows us to write different kinds of programs that we couldn't write before, but AI is still fundamentally a program.

Are you saying the rate of growth will go up? The rate of growth has also stayed roughly constant, right? Only over the last 200, 300 years.

For a while I tried to find AI or look for AI in the GDP curve, and I've convinced myself that this is false.

Yes, my expectation is that it stays in the same pattern.

If a highly experienced smart immigrant came to the country, you wouldn't need to figure out how we integrate them in the economy. They figure it out. They could start a company, they could make inventions, or increase productivity in the world

We have examples, even in the current regime, of places that have had 10-20% economic growth. If you just have a lot of people and less capital in comparison to the people, you can have Hong Kong or Shenzhen or whatever with decades of 10% plus growth

The crucial thing with the Industrial Revolution was that it was not magical. If you just zoomed in, what you would see in 1770 or 1870 is not that there was some key invention. But at the same time, you did move the economy to a regime where the progress was much faster and the exponential 10x'd

I am surprised that it evolved. I find it fascinating to think about all the worlds out there.

The evolution of intelligence intuitively feels to me like it should be a fairly rare event.

It's still surprising to me, intuitively, that it developed.

It's so hard to tell with any of this stuff.

A lot of the smaller models, or the dumber models, remarkably resemble a kindergarten student, or an elementary school student or high school student. Somehow, we still haven't graduated enough where this stuff can take over. My Claude Code or Codex, they still feel like this elementary-grade student.

I don't think they can create culture because they're still kids. They're savant kids. They have perfect memory of all this stuff. They can convincingly create all kinds of slop that looks really good. But I still think they don't really know what they're doing

Self-driving is very interesting because it's definitely where I get a lot of my intuitions because I spent five years on it. It has this entire history where the first demos of self-driving go all the way to the 1980s. You can see a demo from CMU in 1986.

For some kinds of tasks and jobs and so on, there's a very large demo-to-product gap where the demo is very easy, but the product is very hard. It's especially the case in cases like self-driving where the cost of failure is too high.

For example, in software engineering, I do think that property does exist. For a lot of vibe coding, it doesn't. But if you're writing actual production-grade code, that property should exist, because any kind of mistake leads to a security vulnerability or something like that.

In self-driving, if things go wrong, you might get injured. There are worse outcomes. But in software, it's almost unbounded how terrible something could be. I do think that they share that property.

One thing I will almost instantly push back on is that this is not even near done, in a bunch of ways that I'm going to get to.

It's very interesting to hear you say that, that the safety guarantees you need from software are not dissimilar to self-driving.

What people will often say is that self-driving took so long because the cost of failure is so high. A human makes a mistake on average every 400,000 miles or every seven years.

In fact, in terms of wall clock time, it would be much less than seven years because you're constantly outputting code like that.

Self-driving is just one of thousands of things that people do. It's almost like a single vertical, I suppose. Whereas when we're talking about general software engineering, it's even more... There's more surface area.

If somebody's waving down the road this way, you don't need to train for it. The thing will have some understanding of how to respond to something like that.

self-driving is a very nice analogy that you can borrow things from. What is the equivalent of a cone in the car? What is the equivalent of a teleoperating worker who's hidden away and all the aspects of it.

I'm only sounding pessimistic because when I go on my Twitter timeline, I see all this stuff that makes no sense to me. There's a lot of reasons for why that exists. A lot of it is honestly just fundraising. It's just incentive structures. A lot of it may be fundraising. A lot of it is just attention, converting attention to money on the internet, stuff like that.

I've heard many, many times over the course of my 15 years in AI where very reputable people keep getting this wrong all the time.

I'm actually optimistic. I think this will work. I think it's tractable.

For example, Claude Code or OpenAI Codex and stuff like that didn't even exist a year ago. Is that right? This is a miraculous technology that didn't exist. There's going to be a huge amount of demand, as we see the demand in ChatGPT already and so on.

I don't know that there's overbuilding. I think we're going to be able to gobble up what, in my understanding, is being built.

I want this to be properly calibrated, and some of this also has geopolitical ramifications and things like that with some of these questions. I don't want people to make mistakes in that sphere of things.

I feel some amount of determinism around the things that AI labs are doing.

I feel like I could help out there, but I don't know that I would uniquely improve it. My personal big fear is that a lot of this stuff happens on the side of humanity, and that humanity gets disempowered by it.

So what are you working on there? The easiest way I can describe it is we're trying to build the Starfleet Academy.

It has to be rewired and changed to some extent. I still think that we're pretty early.

It's helpful, but it still feels to me a bit like slop. I'd like to do it properly, and I think the capability is not there for what I would want.

Instantly from a very short conversation, she understood where I am as a student, what I know and don't know. She was able to probe exactly the kinds of questions or things to understand my world model.

No LLM will do that for you 100% right now, not even close. But a tutor will do that if they're good.

At the current capability, you don't. That's why I think it's not actually the right time to build this kind of an AI tutor.

Even today, I would say ChatGPT is an extremely valuable educational product.

I'm building the first course. I want to have a really, really good course, the obvious state-of-the-art destination you go to to learn, AI in this case.

In my mind, education is the very difficult technical process of building ramps to knowledge.

Over time maybe some of the TAs can become AIs. You just take all the course materials and then I think you could serve a very good automated TA for the student when they have more basic questions or something like that.

But I think you'll need faculty for the overall architecture of a course and making sure that it fits.

No, I will hire faculty because there are domains in which I'm not an expert. That's the only way to offer the state-of-the-art experience for the student ultimately.

When I'm building Starfleet Academy, I do probably imagine a physical institution, and maybe a tier below that a digital offering that is not the state-of-the-art experience you would get when someone comes in physically full-time and we work through material from start to end and make sure you understand it.

I think you're basically inventing college from first principles for the tools that are available today and just selecting for people who have the motivation and the interest of really engaging with material.

There's going to have to be a lot of not just education but also re-education. I would love to help out there because the jobs will probably change quite a bit. For example, today a lot of people are trying to upskill in AI specifically.

Motivation-wise, before AGI motivation is very simple to solve because people want to make money. This is how you make money in the industry today. Post-AGI is a lot more interesting possibly because if everything is automated and there's nothing to do for anyone, why would anyone go to a school? I often say that pre-AGI education is useful. Post-AGI education is fun.

Education will play out in the same way. You'll go to school like you go to the gym.

People will be able to go much further. Anyone will speak five languages because why not? Because it's so trivial. Anyone will know all the basic curriculum of undergrad, et cetera.

It's going to make learning anything trivial and desirable, and people will do it for fun because it's trivial.

I really do care about humanity. Everyone has to just be superhuman in a certain sense. It's still a world in which that is not enabling us to… It's like the culture world, right?

In the long-term, that probably goes away. It might even become a sport. Right now you have powerlifters who go extreme in this direction.

I feel that the geniuses of today are barely scratching the surface of what a human mind can do, I think.

I love learning, even for the sake of learning, but I also love learning because it's a form of empowerment and being useful and productive.

With what’s happened so far with online courses, why haven't they already enabled us to enable every single human to know everything? They're just so motivation-laden because there are no obvious on-ramps and it's so easy to get stuck.

When you do it properly, learning feels good. It's a technical problem to get there.

For a while, it's going to be AI plus human collab, and at some point, maybe it's just AI.

What I did was I used ChatGPT to ask the questions with the paper in the context window. It worked through some of the simple things. Then I shared the thread to the person who wrote that paper or worked on that work. I felt like if they could see the dumb questions I had, it might help them explain better in the future.

it is in 100% of cases that just the narration or the transcription of how they would explain it to you over lunch is way more, not only understandable, but actually also more accurate and scientific, in the sense that people have a bias to explain things in the most abstract, jargon-filled way possible