Daniel Kokotajlo: evaluation

Transcript context

…Yeah. So to me, it doesn’t seem like he is just sitting there being logically omniscient and getting the answer. It seems like he’s a genius, he’s thought about this for years, probably at some point, he heard a couple of Indian words and a couple of European words at the same time and they kind of connected and the light bulb came on. So this isn’t about having all the information in your memory so much as the normal process of discovery, which is kind of mysterious, but seems to come from having good heuristics and throwing them at things until you kind of get a lucky strike. My guess is if we had really good AI agents and we applied them to this task, it would look something like a scaffold where it’s like, think of every combination of words that you know of, compare them. If they sound very similar, write it on this scratch pad here. If a lot of words of the same type show up on the scratch pad, that’s pretty strange, do some kind of thinking around it. And I just don’t think we’ve even tried that. And I think right now if we tried it, we would run into the combinatorial explosion. We would need better heuristics. Humans have such good heuristics that probably most of the things that show up even in our conscious mind, rather than happening on the level of some kind of unconscious processing, are at least the kind of things that could be true. I think you could think of this as like a chess engine. You have some unbelievable number of possible next moves, you have some heuristics for picking out which of those are going to be the right ones. And then gradually you kind of have the chess engine think about it, go through it, come up with a better or worse move, then at some point you potentially become better than humans. I think if you were to force the AI to do this in a reasonable way, or you were to train the AI such that it itself could come up with the plan of going through this in some kind of heuristic-laden way, you could potentially equal humans. I’ll add some more things to that. So I think there’s a long and sordid history of people looking at some limitation of the current LLMs and then making grand claims about how the whole paradigm is doomed because they’ll never overcome this limitation. And then a year or two later the new LLMs overcome that limitation. And I would say that with respect to this thing of “why haven’t they made these interesting scientific discoveries by combining the knowledge they already have and noticing interesting connections?” I would say first of all, have we seriously tried to build scaffolding to make them do this? And I think the answer is mostly no. I think Google DeepMind tried this.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.