David Rosenthal: prediction

Transcript context

…And that was in 2000. Well, one day, in either late 2000 or early 2001—the timelines are a bit hazy here—Google engineer named George Herrick is talking over lunch with Ben Gomes, famous Google engineer who I think would go on to lead search, and a relatively new engineering hire named Noam Shazeer. Now, George was one of Google’s first 10 employees, incredible engineer. Just like Larry Page’s dad, he had a PhD in machine learning from the University of Michigan. Even when George went there, it was still a relatively rare contrarian subfield within computer science. The three of them are having lunch, and George says offhandedly to the group that he has a theory from his time as a PhD student, that compressing data is actually technically equivalent to understanding it. The thought process is if you can take a given piece of information and make it smaller, store it away, and then later reinstantiate it in its original form, the only way that you could possibly do that is if whatever force is acting on the data actually understands what it means. You’re losing information going down to something smaller, and then recreating the original thing. It’s like you’re a kid in school. You learn something in school, you read a long textbook, you store the information in your memory, then you take a test to see if you really understood the material. And if you can recreate the concepts, then you really understand it. Which foreshadows big LLMs today are like compressing the entire world’s knowledge into some number of terabytes, that’s just like the smashed down little vector set—little at least compared to all the information in the world—but it’s that idea. You can store all the world’s information in an AI model in something that is incomprehensible and hard to understand. But then if you uncompress it, you can bring knowledge back to its original form.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.