Daniel Kokotajlo: belief

Transcript context

…But even in that scenario alone, I can imagine bottlenecks like, oh, you had a benchmark and it got reward hacked for what constitutes AI R&D because you obviously can’t have… maybe you would, but is it as good as a human brain? It’s just like such an ambiguous thing you’d have. Right now we have benchmarks that get reward hacked, right? But then they autonomously build new benchmarks. I think what you’re saying is maybe this whole process just goes off the rails due to lack of contact with ground truth outside in the actual world, outside the data centers. Maybe? Again, part of my guess here is that a lot of the ground truth that you want to be in contact with is stuff that’s happening on the data centers, things like how fast are you improving on all these metrics, and you have these vague ideas for new architectures, but you’re struggling to get them working. How fast can you get them working? And then separately, insofar as there is a bottleneck of talking to people outside and stuff, well they are still doing that. And once they’re fully autonomous, they can even do that much faster. You can have all the million copies connected to all these various real world research programs and stuff like that. So it’s not like they’re completely starved for outside stuff. What about the skepticism that, look, what you’re suggesting with this hyper efficient hive mind of AI researchers, no human bureaucracy has just out of the gate worked super efficiently, especially one where they don’t have experience working together. They haven’t been trained to work together, at least yet. And there hasn’t been this outer loop RL on like, “we ran a thousand concurrent experiments of different AI bureaucracies doing AI research and this is the one that actually worked best”. And the analogy I’d use maybe is to humans in the Savannah 200,000 years ago. We know they have a bunch of advantages over the other animals already at this point, but the things that make us dominant today, joint stock corporations, state capacities like this fossil fueled civilization we have that took so much cultural evolution to figure out. You couldn’t just have figured it out in the savannahs like, “oh, if we had built these incentive systems and we issued dividends, then we could really collaborate here” or something. Why not think that it will take a similar process of huge population growth, huge social experimentation, and upgrading of the technological base of the AI society before they can organize this hypermind collective, which will enable them to do what you imagine an intelligence explosion looks like?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.