Ben Gilbert: belief

Transcript context

…My first feeling about the language model was how clever it was to just mask out words and make it predict the next word. It’s self supervised learning at its best. We have all this text. I know what the answer is. I’ll just make you guess it. My first impression of BERT was really how clever it was. Now the question is how can you scale that? The first observation on almost everything is interesting, and then try to understand intuitively why it works. Then the next step is from first principles. How would you extrapolate that? Obviously, we knew that BERT was going to be a lot larger. Now, one of the things about these language models is it’s encoding information. It’s compressing information. Within the world’s languages and text, there’s a fair amount of reasoning that’s encoded in it. We describe a lot of reasoning things. if you were to say that a few step reasoning is somehow learnable from just reading things, I wouldn’t be surprised. For a lot of us, we get our common sense and reasoning ability by reading. Why wouldn’t a machine learning model also learn some of the reasoning capabilities from that? From reasoning capabilities, you could have emergent capabilities. Immersion abilities are consistent with intuitively from reasoning. Some of it could be predictable, but still, it’s still amazing. The fact that it’s sensible doesn’t make it any less amazing. I could visualize literally the entire computer and all the modules in a self-driving car. The fact that it’s still keeping lanes makes me insanely happy. I even remember that from my first operating systems class in college, when I finally figured out all the way from programming language to the electrical engineering classes, bridged in the middle by that OS class. I’m like, oh, I think I understand how the Von Neumann computer works soup to nuts, and it’s still a miracle. Exactly. When you put it all together, it’s still a miracle.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.