Nathan Labenz: prediction

Transcript context

…re basically 33 load bearing forecasts, basically starting from what even happened. Like why did the government issue this export control? Was it a simple misunderstanding? Is this political leverage? This is really about foreigner threat because Fable is actually dangerous for hacking, etcetera. We didn't know those things. So I kind of put it all together and when I looked at all of the outcomes and I talked about it with Claude Code a lot, one thing came out, which is basically every forecast and every scenario. I had thought that access would come to Americans 1st and then foreigners at some later point in the future, and that was wrong. When it came out last week, it came back for everybody, so clearly there was some wait in one of my scenarios that was wrong, but I had a basically like kind of a correlated failure in there somewhere. I still haven't completely understood where my reasoning was wrong. It's also possible I just got really unlucky and the outcome we're in was just extremely unlikely. There's an n = 1. You can never know if anyone forecast is great. That's one of the hard things about it. But I think I systematically got it wrong by having a bunch of correlated reasoning failures across my various scenarios, so this definitely does happen. Metaculous has a system like this. In the years since I was the CTO there, they have built an actual causal graph platform and product. So you can go to the Metaculous site and click right and you'll find it there. I think the field still generally believes that things like this will work, but nobody has actually made a good one before. And I tried my best over basically like, you know, 12 to 16 hours of the Fable situation. I think I made a pretty good model. I think I did. I was close to having a very accurate forecast, but I didn't quite get it. I don't think. I don't think those meticulous models on their website right now are so amazing, but I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried for a long time. When I was the CTO of Metangolis, honestly, it was it was kind of the dream. It was the Holy Grail. Can we tie all of these forecasts together into some sort of causal graph? And I think what I can say is that AI makes this tractable. There was just no way that that was going to work with a bunch of human economists looking at Freddie Mac or Fannie Mae. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed, and I don't think Future Search is the only org that is working on this right now. Before he left the unhedged version. So Future Search contributed some forecast to AI 2027 and we studied that problem pretty seriously with the evidence of a little bit over a year ago. And we built a model of R&D take off speeds under the kind of the core AI 2027 scenario where the main way things get crazy is that AI is used more in the development of AI, first by achieving the superhuman coder milestone and then the superhuman AI researcher milestone. And I am unhappy to report that I think that story is generally correct. re in the development of AI, first by achieving the superhuman coder milestone and then the superhuman AI researcher milestone. And I am unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but I my forecast from that process of leading to something that looks like super intelligence around 2031 is roughly stable. I think the things that have happened in the years since AI 2027 come out very much indicate the theory that the most important thing going on is how useful is AI and improving the productivity of AI researchers within Frontier Labs. I've made public predictions that I thought Anthropic was going to run away with it because they had the best feedback loop of talent and actually using their AI internally. I think that has been, you know, n = 1. But I think it's been totally shown that that's been happening recently. So I think that will continue to happen. And Dan's closing confession about the whole project to prediction markets and a hope for what AI forecasting could still become. Maybe just in closing, sketch out a little bit more of the future as you hope it might unfold. Not necessarily the most likely scenario, because maybe the most likely thing is people act foolishly and don't take advantage of the benefits of forecasting. But like, if we really do a good job right, and we're and we're interested in truth seeking and we get the AIS working as well as you think they might, how do you think life feels different? Yeah, I have to leave with another example of me being a bad forecaster. I, I guess everyone who tries forecasting thinks they're a bad forecaster because they see things getting wrong. Here's a prediction that I made really strongly 5 or 10 years ago that has basically been totally falsified. I predicted that if we had highly visible, highly liquid prediction markets that were covering all of the like the major technological and political and economic things going on, that humanity would be wiser and people would make better decisions in government. So here we are. We have Pauline market and call sheet. ajor technological and political and economic things going on, that humanity would be wiser and people would make better decisions in government. So here we are. We have Pauline market and call sheet. I don't see any wisdom or better decisions coming out of all of that gambling going on on those platforms. So for me, part of the what is our AI future is trying to understand the present a little bit better. Why is having thriving prediction markets not transforming, say, the news or how people learn information or plan for their futures? Again, one simple answer is that it does. It just takes a while. We're only about a year into prediction markets having, you know, major headlines and being seen by everybody. Maybe it just takes a while for people to change their habits. A is, if that's the case, can move much faster as a is get better at forecasting. I guess ultimately you said this, Nathan, like we're after the epistemics. Like it's not necessarily just forecasting like predicted this outcome. We want models that are reasonable. And one of the beautiful things about forecasting as a human practice is it makes you more epistemically virtuous. The more that you try to forecast and actually write down what you get wrong and do these post mortems, the more it humbles you and it makes you more open minded. It makes you more of a fox instead of a hedgehog. It just makes you like a more like reasonable person. And so prediction markets with all these people doing this should be leading to people being more reasonable. Again, I think people aren't doing a whole lot of forecasting on prediction markets. They're doing a lot of trading and a lot of gambling, which are related to forecasting but not forecasting. If the AIS get more, if they get better at forecasting and they become better epistemically, then we could be in a world where just talking to a chat bot, you were getting something so much wiser and more grounded and more honest about it's uncertainty and more poking about you and your own uncertainties as the person talking to the chat bot. And that I think could make an absolutely enormous difference. I think again, putting my kind of cold blooded forecasting hat back on, I think that the technological outcomes of AGI will come before the cultural change happens. So I'm very much on the the AI safety camp. I really think we should slow things down, give us more time. We should fund more AI safety research and do more policy because if we have time for the wisdom of having these alien intelligences around helping us, if we can leverage them and actually make better decisions before the critical decisions get made, there's going to be a series of decisions in the 21st century that we're going to look back on, like this decisions made in the 20th century about communism and World War 2 and the atom bomb and all of those things. Those decisions are coming. Maybe some of them have already been made. Those decisions as of right now, I don't think are very well informed by like very rigorously epistemic accurate forecasting AIS.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.