Dwarkesh Patel: evaluation

Transcript context

…I actually have the same hypothesis I had even all the way back in 2017. I think I talked about it last time, but I wrote a doc called “The Big Blob of Compute Hypothesis”. It wasn’t about the scaling of language models in particular. When I wrote it GPT-1 had just come out. That was one among many things. Back in those days there was robotics. People tried to work on reasoning as a separate thing from language models, and there was scaling of the kind of RL that happened in AlphaGo and in Dota at OpenAI. People remember StarCraft at DeepMind, AlphaStar. It was written as a more general document. Rich Sutton put out “The Bitter Lesson” a couple years later. The hypothesis is basically the same. What it says is that all the cleverness, all the techniques, all the “we need a new method to do something”, that doesn’t matter very much. There are only a few things that matter. I think I listed seven of them. One is how much raw compute you have. The second is the quantity of data. The third is the quality and distribution of data. It needs to be a broad distribution. The fourth is how long you train for. The fifth is that you need an objective function that can scale to the moon. The pre-training objective function is one such objective function. Another is the RL objective function that says you have a goal, you’re going to go out and reach the goal. Within that, there’s objective rewards like you see in math and coding, and there’s more subjective rewards like you see in RLHF or higher-order versions of that. Then the sixth and seventh were things around normalization or conditioning, just getting the numerical stability so that the big blob of compute flows in this laminar way instead of running into problems. That was the hypothesis, and it’s a hypothesis I still hold. I don’t think I’ve seen very much that is not in line with it. The pre-training scaling laws were one example of what we see there. Those have continued going. Now it’s been widely reported, we feel good about pre-training. It’s continuing to give us gains. What has changed is that now we’re also seeing the same thing for RL. We’re seeing a pre-training phase and then an RL phase on top of that. With RL, it’s actually just the same. Even other companies have published things in some of their releases that say, “We train the model on math contests — AIME or other things — and how well the model does is log-linear in how long we’ve trained it.” We see that as well, and it’s not just math contests. It’s a wide variety of RL tasks. We’re seeing the same scaling in RL that we saw for pre-training. You mentioned Rich Sutton and “The Bitter Lesson”. I interviewed him last year, and he’s actually very non-LLM-pilled. I don’t know if this is his perspective, but one way to paraphrase his objection is: Something which possesses the true core of human learning would not require all these billions of dollars of data and compute and these bespoke environments, to learn how to use Excel, how to use PowerPoint, how to navigate a web browser. The fact that we have to build in these skills using these RL environments hints that we are actually lacking a core human learning algorithm. So we’re scaling the wrong thing. That does raise the question. Why are we doing all this RL scaling if we think there’s something that’s going to be human-like in its ability to learn on the fly? I think this puts together several things that should be thought of differently. There is a genuine puzzle here, but it may not matter. In fact, I would guess it probably doesn’t matter. There is an interesting thing. Let me take the RL out of it for a second, because I actually think it’s a red herring to say that RL is any different from pre-training in this matter. If we look at pre-training scaling, it was very interesting back in 2017 when Alec Radford was doing GPT-1. The models before GPT-1 were trained on datasets that didn’t represent a wide distribution of text. You had very standard language modeling benchmarks. GPT-1 itself was trained on a bunch of fanfiction, I think actually. It was literary text, which is a very small fraction of the text you can get. In those days it was like a billion words or something, so small datasets representing a pretty narrow distribution of what you can see in the world. It didn’t generalize well. If you did better on some fanfiction corpus, it wouldn’t generalize that well to other tasks. We had all these measures. We had all these measures of how well it did at predicting all these other kinds of texts. It was only when you trained over all the tasks on the internet — when you did a general internet scrape from something like Common Crawl or scraping links in Reddit, which is what we did for GPT-2 — that you started to get generalization. I think we’re seeing the same thing on RL. We’re starting first with simple RL tasks like training on math competitions, then moving to broader training that involves things like code. Now we’re moving to many other tasks. I think then we’re going to increasingly get generalization. So that kind of takes out the RL vs. pre-training side of it. But there is a puzzle either way, which is that in pre-training we use trillions of tokens. Humans don’t see trillions of words. So there is an actual sample efficiency difference here. There is actually something different here. The models start from scratch and they need much more training. But we also see that once they’re trained, if we give them a long context length of a million — the only thing blocking long context is inference — they’re very good at learning and adapting within that context. So I don’t know the full answer to this. I think there’s something going on where pre-training is not like the process of humans learning, but it’s somewhere between the process of humans learning and the process of human evolution. We get many of our priors from evolution. Our brain isn’t just a blank slate. Whole books have been written about this. The language models are much more like blank slates. They literally start as random weights, whereas the human brain starts with all these regions connected to all these inputs and outputs. Maybe we should think of pre-training — and for that matter, RL as well — as something that exists in the middle space between human evolution and human on-the-spot learning.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.