Steven Pinker: uncertainty

Transcript context

…Are you discouraged that linguistics has had so little impact on current large language models? Yes, I am. I don’t know whether it’s that linguists themselves have not made themselves useful enough, have not been part of the conversation, or the cynical view is linguistics has just been on the wrong track and it has nothing to contribute. I tend to think more of the former. There are some features in large language models that actually implicitly build in some of the insight of linguistics. The fact that you’ve got all of these hidden layers is an acknowledgment that the regularities in language and the regularities in thought that language expresses are abstractions. They’re not triggered by individual words, but they’re patterns and patterns and patterns, groupings, and larger and larger groupings. The fact that one of the breakthroughs in this generation of large language models is what I call attention heads, which are basically processors that look at temporally separated words in search of regularities, as opposed to adjacent ones. The fact that they’re not trained on the words themselves, but on tokens that correspond to more or less to morphemes. The fact that they’re trained on embeddings, which is not, again, not the words themselves, but some backhanded indirect representation of meaning. In that case, the context that a word typically occurs in. There are a lot of discoveries of linguistics that are implicitly built in, but linguistic theory in the sense of what you see in a journal like Linguistic Inquiry, don’t have direct counterparts. Maybe they wouldn’t need all of those gigawatts and all of those GPUs if they were able to cut to the chase quicker by building in linguistic insights. My hunch is yes, but the models are so complex that I can’t say for sure. Does it favor Chomsky? Does it favor the view that language is simpler than we thought? Where does it push your intuitions?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.