Keith Duggar: belief

Transcript context

…Yeah. And this is the I mean, I think we're getting to putting the pin in in the center of the dartboard here. So he said the the problem is that neural networks have too many degrees of freedom. They're like a pile of sand. Right? And we are different from that because, you know, the the very basis of how we think is correlated to how the world works. So there's something about building up, you know, you said building up not not tearing down. And and he said it's not about what you know, it's about how you got there and how evolvable the knowledge is. So where you can go with that knowledge. And it's not necessarily that stochastic gradient descent is a bad thing in of it itself. It's simply that it's statistically intractable to use anything other than stochastic gradient descent. Otherwise, it simply doesn't work. So he's not necessarily saying, I mean, maybe he is, but he's saying that these evolutionary algorithms are really, really good when you're building up, when you have a very, very sparse search space. And and then the other thing which is very important which we didn't get to is this whole building up thing like the neat algorithm and you can bring that in. Is this idea that we need to have a form of training or learning about the world which monotonically increases information and complexity. So we're we're adding this thing and we're adding this thing. And every time we compose things together, we're looking at the evolvability. So we're not just saying, can you do the thing now? We're saying, is it actually likely to be able to deal with future things that I might encounter? When children are born, they still have 2 legs, basically. So you shouldn't just think of it as a random, you know, crossover operation. It's an operation that respects the the provenance, the topology, the structure. Right? So there are, you know, in practice, there are crossovers and mutations within certain topological frames. But some things are held constant because they need to be held constant. But on your comment about degrees of freedom, I think Kenneth was saying that we actually want to reduce degrees of freedom. So he's not kind of saying, oh, we just need degrees of freedom for stochastic gradient descent, but degrees of freedom are okay. I think he was saying that there's actually a magical Goldilocks zone when it comes to degrees of freedom. So we want the degrees of freedom to be at least a representation of how the world works. But necessarily, there should actually be more flexibility than how the world works. But if there's too much flexibility, you get a weird kind of mode collapse and you don't see the emergence of evolution. Degrees of freedom, the way I think about is, like, it's like a memory with an algorithm. You want to use as much as necessary and no more. And it's really hard to it's really hard to figure that out. Yeah. I mean, just imagine Einstein. Of course, there's an element of serendipity. Right? You have to be in the right place in the right time. But there are still just mental degrees of freedom. And simply, if there are too many degrees of freedom, then relativity wouldn't be wouldn't be conceived of. Right? You you need to have that that that spark.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.