Ryan Greenblatt: evaluation

Transcript context

…But I feel like one effect will be that we will have gotten rid of all the low-hanging fruits by 2030. I feel like scaling laws will have been, in math history, like Descartes finding the Cartesian grid and doing very basic mathematics. Eventually, if we want to keep making progress in the 2030s, it’s going to be like doing whatever bullshit is happening at the frontiers of mathematics right now. That could be right. My sense is that some domains are structurally different in terms of how they operate and how much they depend on deep abstractions. Physics and math are much more on the side of being very far on the deep, hard-to-come-up-with-ideas side, whereas I think ML and most other domains are much more amenable to hill climbing. That’s my sense of how this will go in the future. Even in the regime where your AIs are having to plow — it’s 2030, a bunch of low-hanging fruit in research has already happened, and they need to make further progress — I still suspect that a bunch of the work will live more on the side of building increasingly complicated infrastructure and having really good intuition about what the experiments roughly look like. So I’m probably less sympathetic to the idea that the thing the AIs will lack is some deep insight. I’m more sympathetic to the idea that they really need a bunch of taste about in-the-weeds experiments that they currently don’t have. They need a bunch of intuition for what sorts of training approaches would work and what wouldn’t, in ways that current researchers have. Even in cases where there has been some breakthrough in AI, oftentimes in retrospect it looks like a big bottleneck to making that breakthrough happen was getting all of the micro details and mungy intuition right. An example of this is training AIs to be good at reasoning and chain of thought, doing RL on chain of thought. It looks like you probably could have done RL and chain of thought on GPT-3 and gotten kind of interesting results on math if you had really scaled it up and done a good job. But at the time, there was low-hanging fruit. Also, doing a good job with that training is kind of in the weeds on all the technical implementation and scaling it up and getting the hyperparameters right. So maybe you can demonstrate everything on Qwen 1B or whatever and get some sense that this whole thing is going to work. But people didn’t demonstrate it as early as they could have because of all of these other mungy details and intuition about exactly how to tune the parameters and how to set things up. This is my remaining skepticism, honestly, about this story. I’m not sure I understand why, if research breakthroughs are so amenable to intelligence, AI progress has not been historically faster than it could have been. As you were saying, by the time RLVR actually worked — even though you could have done it with less compute — we had to wait for oceans of compute, gigawatts of compute, to be available before people were doing this training, on the trajectory of compute continuing to increase so we make more breakthroughs. I don’t know. I feel like there were a lot of AI researchers in the year 2022 who were trying to crack reasoning. Was it just that they were bottlenecked by the ability to write infrastructure code, or what was happening?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.