Yi Ma: evaluation

Transcript context

…coexist, we believe. At least those are the 2 principles, parsimony and the consistency or self consistency, are actually the 2 characteristic about how our memory works. So we want to have understanding which carves the world up by the joints, which represents the important invariances in the world. And the thesis is, I think, that compression might be necessary for understanding. My possible concern with that is that what we are doing with machine learning is representing extant examples of a long phylogenetic tree of evolution. Mhmm. So to what extent does knowing their representation now help us? Do we also need to know how they evolved and where they might go in the future? The the process to acquire knowledge, to gain information about our side world, that's a compression. Find what is compressible. What has orders? What phenomena has orders? Has low dimensional structures that allow us to predict, to rule out variabilities, to predict world tomorrows, or predict world better, in essence. So in a sense that that is the ability we believe is really what intelligence is all about, at least the common intelligence we're talking about, right? We can talk about the more higher level intelligence later. And if you look at the history of life, how life was developed, So we actually come to believe, right? You know the mechanism that we laws that governs the physical world, we call it physics, right? But what is the mechanism that governs the evolution of life? I think it's intelligence, right? Even the process you mentioned that through the evolution and life evolves, precisely they learn more and more knowledge about the world, and they encode them through DNAs to pass it on to next generation. And that is a compressing, that's a process to compress knowledge that learn about the world through our DNAs. But the mechanism to update it is actually very brutal, very brute force. And through, you know, random mutation and the natural selection. Yes, it does evolve. It does advance, but at a huge cost of resource, time, and also very unpredictable. Which if you're acute, you probably observe there's some similarity with how current big model evolves, right? Many, many groups try without principle, trial and error, empirical. And the lucky ones survive and gets advocated everywhere and become very, very popular, right? Dominate the practice. So in a sense that it can make it an allergy, right? I think to the people, students ask me at which stage our artificial intelligence is at today. Then there's already an allergy in nature, right? We are very much at the early stage of the life form, right? And so hence, that is a compression process. That's a process that also gain knowledge about the world. But of course, on, we develop individual animals to develop the brain, develop neural systems, develop senses, including visual and touch and so on. So we actually start to use a very different mechanism to learn, to compress our observations, to learn knowledge, and to build memories of Oxford world. And even individuals start to have that ability, rather than just inherit knowledge from their DNAs. So that's a different stage. Then that part of the knowledge is no longer encoded in our genetics, in our genes, but also in our brains. And that's actually a level of intelligence we talk about most the time these days, you know, which is common to animals, which is common to humans, and the knowledge or the intelligence we talk about what brain functions. Yeah, mean, think we would definitely agree with the statement that intelligence as a system produces artifacts. So Charle's example is a road building network. It produces roads, and the system has adaptivity because it can create new routes where they weren't there before. And then there's the question of, well, there are many ways to compress a thing. Uh-huh. So some ways of compression represent the world at a deep abstract level, and some don't. So we might argue that LLMs today, even though they do compress the data, they only compress it in a superficially semantic way. Mhmm. And then there's this notion of, well, maybe we agree that intelligence is about the synthesis of new knowledge. So it's the acquisition of new knowledge, but we can only do that if the knowledge we already have represents the world at a deep abstract level. So rather than it being random mutations in evolution, it's very, very structured because the processes are physically instantiated, which means rather than just doing something completely random, it's guided by the process which created the manifold hypothesis comes to mind, which is this idea that all natural data falls on some low dimensional, you know, structure, you know, with with a low intrinsic dimension. And the other thing that springs to mind is I mean, I'm I'm a fan of geometric deep learning, which is this idea of, you know, we should imbue inductive priors in the system, which represents symmetries and and geometric structures in in the world. And and I think as a principle, that's deeply embedded in in this idea. Exactly.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.