Luke Drago: recommendation

Transcript context

…We're facing this tension between trying to control the downsides of AI by centralizing it and then spreading the upside by giving as many people as possible access to the models. So one answer to this tension is just to say that we need to open source AI fully. What do you think about that vision and how does it interface with what you're talking about? So I am probably more like pro open source than I think the average person on the podcast. And I think part of this is because of this real fear of monopolization. I think it is the case that if open weights models are not a core part of the future, that you can increasingly charge these wild rents for them. I think there are a couple people who have strong incentives to build them. So I don't think it's the case that like I don't think it's the case that they're going to fall behind in some near future. And I also think there's this very pervasive argument, I think, especially within the AI safety community, that open weights models are always going to be behind. It is absolutely true that in a hard takeoff scenario where you just foom and go straight to superintelligence, that that's going to be the case. Someone's going to win that race. That's game over. In basically every other scenario, what we have seen is the exact opposite. I remember hearing a couple years ago that there's like no way that open weights models could catch up. The 2 behind, and especially like there's no way that China could catch up. It's just impossible. Chinese models right now, Chinese open weights models are like six months behind the frontier, and some of them I think maybe are even more ahead. Kimi-K2, for example, is a really excellent English writing model. I would wager it's probably the state-of-the-art at that. This does not look like that we are seeding, open weights models are slowing down, the gap continues to close, even on providers that have less access to high-quality computes. There's something going on in both the way in which we train them and the data that we're using that still provides advantages such that compute isn't everything. And so I think if the argument that I oftentimes hear is like, open weights can't catch up, it's not a core part of the story, I just don't think this is true. I think if you're taking AI safety seriously, you're going to have to focus on making open weights models safe because open weights models are going to be a reality and they're going to be quite powerful. How do we do that though? I guess that's the main worry with open weights models. This is just we can't If we put something out there that's open weights, we can't then take it back. Exactly. So we don't have this feedback loop of trying to test something and then pulling back and then perhaps putting a more limited version of that model out there. So how do we deal with the technology where if we release it, that capability suite is now out there indefinitely?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.