Daniel Kokotajlo: evaluation

Transcript context

…I don't think I have. Okay. Well, we can I'll briefly mention that before getting into the benchmarks plus gaps thing. So the the bio anchors framework, it's called bio anchors because it references the human brain. And I think that part's actually the less exciting and plausible part of it. The part that I think is is more robust and more worth using is this core idea that you can can think of this trade off between more time to come up with new ideas and do AI research and, more compute with which to do the AI research. And you can sort of think you can you can make a big 2 2 dimensional plot. And you can imagine, Okay, 10 more years, 20 more years, 30 more years. How does the probability that we get to AGI go up with more time? But you can also imagine not more time, but just more compute. Could we get to AGI today if we had 5 orders of magnitude more compute, 10 orders of magnitude more compute, 30 orders of magnitude more compute? Insight there is that the answer is, yeah, probably. Like for example, if you had 10 to the 45 floating point operations, you could do a training run that's basically just simulating the entire planet Earth and all life evolving on it for a billion years with that amount of compute. And the thought there is that you don't really need to understand how intelligence works at all if you're building it with that type of training run because there's no insight coming from you. You're just letting nature do its thing and letting evolution take its course. And so the thought is that we can make a not guaranteed, but like a soft upper bound at something like 10 to the 45. And then you can make other sort of soft upper bounds. You can think, well, what could we do with 10 to the 36? And you can lay I I wrote a blog post about this in 2021. You know, suppose we had 10 to the 36 flop. What are some, like, really huge types of training runs we could do? And then, like, what's our guess as to how likely that is to work? And and what you can do is you can sort of you can start to smear out your probability mass over this dimension of compute. And so you sort of have a soft upper bound, and then you have, of course, a lower bound, which is the amount that we already have done. You know, we we clearly haven't done it right now with this amount of compute. And so that gives you this smeared probability distribution over compute. And then you think, okay. But now we're also gonna get new ideas. And so as new ideas come along, we're going to be able to come up with more efficient methods that allow us to train it with less compute. So you can think of your probability distribution as shifting downwards while also the amount of actual compute increases. And then that gets you your actual distribution over years. And I think this is the right sort of basic framework for calculating these sorts of timelines. also the amount of actual compute increases. And then that gets you your actual distribution over years. And I think this is the right sort of basic framework for calculating these sorts of timelines. But it's a sort of relatively abstract, like, low information framework that doesn't really look at the details of the technology today and the details of the benchmarks. So I think it's a good way to get your prior, so to speak. But then you should update based on actual trends on the benchmarks and so forth, which is what I'm about to get to. But the the reason why I mentioned this prior process is that 10 to the 45 floating point operations isn't actually that far away from where we are right now. Right now, we're at, like, what, like 10 to 26 or something for for training runs. And we're gonna be crossing a few orders of magnitude, in the next couple of years. And so even if you just had like a an in even even if you just smeared out your probability mass with maximum uncertainty across the, like, orders of magnitude from where we are now to 10 to the 45, there'd be a non negligible amount that it's gonna happen in the next few years. And so even on priors, you should think it decently plausible that it could happen by the end of the decade. And then you should update your prior based on the actual evidence, which I'll now get to. So the actual evidence, I would say, let's look at agentic coding benchmarks. That seems to me to be, the most informative thing to look at. And the reason for that is because I don't think that the fastest way to get to superintelligence is in a single leap where humans come up with the new paradigms, in their own brains. I think rather it's going to be this more gradual process where humans automate more of the AI research process, and then that gets us to the new paradigms fast. And so I think that the the the lowest hanging fruit as far as the AI research process is concerned that's going to automate first is the coding. So I'm looking to see when will we get to the point where the the coding is basically all handled by LLM like AI assistants. And we have benchmarks for that, sort of. Like, you know, come places like METR, METR, have been, building these little coding environments, doing all these coding tasks. The the companies themselves have been doing this, of course, because they are racing as fast as they can to get to this automated coder milestone. And, so we look at those, we extrapolate trends on them. And we we forecast that, well, in the next couple of years, they're basically gonna saturate. We're gonna have AIs that can just crush all of these coding tasks. And they're relatively they're they're not something to scoff at. They're not just multiple choice questions. They're, like, the sort of task that would take a human, like, 4 hours to do or 8 hours to do. But that's not the same thing as completely automated coding. So first we extrapolate to when they saturate the benchmarks, and then we try to make our guess as to what the gap is between the first system that can completely saturate these benchmarks and the first system that can actually automate the coding.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.