Ben Gilbert: observation

Transcript context

…Wow. This is 2015? This is two years before the Transformer. Karpathy is at OpenAI. He both comes up with the idea or espouses the idea of a chat bot, so that had already been discussed. But even before we had the Transformer, the method to actually pull this off, he had the idea that—and there’s an important part here—it figures out the pattern in how words follow each other in sentences. There’s this idea that the very structure of language and the way to interpret knowledge is actually embedded in the training data itself rather than requiring labeling. This is so cool. So at Spring GTC this year, Jensen did a fireside chat with Ilya. It’s amazing. You should go watch the whole thing. In it, this question comes up. Jensen poses as a straw man, like hey, some people say that GPT-3, -4, ChatGPT, everything going on, all these LLMs, they’re just probabilistically predicting the next word in a sentence. They don’t actually have knowledge. Ilya has this amazing response to that. He says, “Okay, well consider a detective novel. At the end of the novel, the detective gathers everyone together in a room and says, I am now going to tell you all the name of the person who committed the crime. And that person’s name is blank.” The more accurately an LLM predicts that next word, i.e., the name of the criminal, ipso facto, the greater its understanding not only of the novel but of all general human-level knowledge and intelligence. Because you need all of your experience in the world and as a human, to be able to guess who the criminal is. The LLMs that are out there today, GPT-3, GPT-4, Llama, Bard, these others, can guess who the criminal is.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.