Ryan Greenblatt: belief

Transcript context

…So maybe let’s be more concrete. Here’s what I think. My claim is that if you went back to 2022 and you had GPT-3.5, and you were trying to make it better at coding without human experts, I think it would have just been very, very difficult. Let me give you an example of what I imagine would be the difficulty of going from GPT-8 to ASI. One of the things you’d want ASI to be good at is: I’m going to take over a company and make it much more profitable and do all kinds of crazy shit to make it work better. I’m going to take over a fab and produce more chips. I’m going to go into Congress and try to convince them to pass some bill, et cetera. This is what I imagine five more years of AI progress at this pace would enable an AI to be able to do. This is the thing I’m really worried about: ASI that can understand how to do crazy shit in the world, that can do what Kissinger can do, can do what Steve Jobs can do, et cetera, and also his engineers and so on. I’m not sure how you get that without the relevant world data, which is the equivalent of Mythos being really good at coding while not having the coding environments that have improved it relative to GPT-3. Here are a few points. First, I bet if you look at randomly sampled training environments for Mythos, they’re actually very different from what it looks like to actually use the model in practice. My sense is that the RL distribution has really large deviations from the real-world data distribution, and it’s significantly smoothed over by a mix of transfer and having a small amount of data focused on the real world. My sense is that this will be a similar mechanism as how it works for the crazy, wildly, quite superhuman AI you get as a result of five years of AI progress on top of fully automated AI R&D. So let’s go through this a little bit. In particular, I think that you could train an AI to be really, really good at learning on the fly and doing something analogous to in-context learning, but potentially using somewhat different mechanisms, in a wide variety of RL environments. You build all these different RL environments where the AI has to adapt on the fly, learn on the fly, figure out what it should do, understand its situation better, and learn really quickly from feedback in order to succeed at its objective. And it has things like limited resources, and if it messes up, it can end up in a much worse position. If you train on a huge number of these environments, you will learn general skills of picking up context on the fly, and we’re already seeing this. It’s already the case that AIs are now much better at understanding roughly what’s going on and picking up context from a limited amount of information they’re given access to. Then those AIs could be put on the job at TSMC. Even though TSMC is not literally in their data distribution, their data distribution is really wide, and the AIs are extremely good on their data distribution, such that it transfers to picking up being good at being an engineer at TSMC and learning that on the fly. The way the AI gets good at being a TSMC engineer isn’t that it has a ton of cached knowledge on being a good TSMC engineer. It’s that it does the equivalent of some scaled-up version of in-context learning there. That’d be the most prosaic story. Obviously, there’s a bunch of different ways this could go. I think this maybe comes down to a difference of intuition about how far you can get. When I think about really smart people I know, they’re just not that effective in domains they don’t understand that well.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.