Scott Alexander: belief

Transcript context

…I will actually disagree with this. We know that humans can do like, we have examples of humans doing this. I agree that we don’t have logical omniscience because there is a combinatorial explosion, but we are able to leverage our intelligence to… one of my favorite examples of this is David Anthony, the guy who wrote the Horse, the Wheel and Language. He made this super impressive discovery before we had the genetic evidence for it, like a decade before, where he said, look, if I look at all these languages in India and Europe, they all share the same etymology. I mean literally the same etymology for words like ‘wheel’ and ‘cart’ and ‘horse’. And these are technologies that have only been around for the last 6,000 years, which must mean that there was some group that these groups are all, at least linguistically, descended from. And now we have genetic evidence for the Yamnaya, which we believe is this group. You have a blog where you do this. This is your job, Scott! So why shouldn’t we hold the fact that language models can’t do this more against them? Yeah. So to me, it doesn’t seem like he is just sitting there being logically omniscient and getting the answer. It seems like he’s a genius, he’s thought about this for years, probably at some point, he heard a couple of Indian words and a couple of European words at the same time and they kind of connected and the light bulb came on. So this isn’t about having all the information in your memory so much as the normal process of discovery, which is kind of mysterious, but seems to come from having good heuristics and throwing them at things until you kind of get a lucky strike. My guess is if we had really good AI agents and we applied them to this task, it would look something like a scaffold where it’s like, think of every combination of words that you know of, compare them. If they sound very similar, write it on this scratch pad here. If a lot of words of the same type show up on the scratch pad, that’s pretty strange, do some kind of thinking around it. And I just don’t think we’ve even tried that. And I think right now if we tried it, we would run into the combinatorial explosion. We would need better heuristics. Humans have such good heuristics that probably most of the things that show up even in our conscious mind, rather than happening on the level of some kind of unconscious processing, are at least the kind of things that could be true. I think you could think of this as like a chess engine. You have some unbelievable number of possible next moves, you have some heuristics for picking out which of those are going to be the right ones. And then gradually you kind of have the chess engine think about it, go through it, come up with a better or worse move, then at some point you potentially become better than humans. I think if you were to force the AI to do this in a reasonable way, or you were to train the AI such that it itself could come up with the plan of going through this in some kind of heuristic-laden way, you could potentially equal humans. I’ll add some more things to that. So I think there’s a long and sordid history of people looking at some limitation of the current LLMs and then making grand claims about how the whole paradigm is doomed because they’ll never overcome this limitation. And then a year or two later the new LLMs overcome that limitation. And I would say that with respect to this thing of “why haven’t they made these interesting scientific discoveries by combining the knowledge they already have and noticing interesting connections?” I would say first of all, have we seriously tried to build scaffolding to make them do this? And I think the answer is mostly no.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.