Keith Duggar: disagreement

Transcript context

…No. But okay. I understand your question. But tensor logic does allow that. You know, step 1, you can you can encode a multilayer perceptron, the entire multilayer perceptron, and I do that in the paper with a single tensor equation. All the layers provided that they all use the same nonlinearity can be encoded in 1 equation. Okay? Number 1. You can also have different equations for different layers or typically sets of layers, you know, to your However way you please. But from the point of view of structure discovery, the thing to realize is that if you create, if you set up 1 of these very general equations that you can in tensor logic, that in some sense it can, it's a very broad classes of architecture, then what the learning does is it discovers the architecture within that space. Right? Which if you think about it at some level is what a neural network, when you compare an ordinary multilayer perceptron with a set of rules, right? A multilayer perceptron is you you can take in fact there was a system called k band in the early days that did this, very clever. It initialized a a a a multilayer perceptron with a set of rules because each neuron is a rule. Right? But it's also more flexible because now you can have weights. Right, but but if but you know, neuron, a single neuron can represent a conjunction and therefore a layer can represent a junction and so forth. So when you're learning weights in an ordinary neural network, you can actually see it as learning the structure of a set of rules. What tensor logic is doing, this is at a more powerful level, like like that was just propositional and now this this is at the full level of generality of of first order logic. But you can learn the structure and then of course then there's more than 1 way to do that, and you can also decide how black and white you want the structure to be, what you want to leave as weights, what you want to discretize. But the structure itself can be learned by taking a tensor equation. A tensor equation is a very general thing, right? When when you learn the weights of those tensors that the that materializes to a specific network structure. Yeah. So I I I understand that. Let me bring this back to like to the folks who are familiar with, you know, PyTorch or traditional techniques. What you described is, yeah, I can just create a fully connected, you know, network with however many layers I want and then let SGD, you know, find all the weights. That doesn't work. Like, it doesn't work in practice, and it's not gonna work with tensor logic. You know, it's just a different representation of the same fundamental problem, which is there's too many degrees of freedom. It's not gonna learn anything useful. This is why so much alchemy goes into structuring, you know, constrained networks to have certain, you know, built in, you know, inductive bias. Right? No. Absolutely. So to take another example, you can also do an entire ConvNet in just 1 tensor equation. And and, you know, like, the quintessential example of, like, yes, complete connections don't work is a multilayer perceptron for for vision, right, which you replace with a convnet that actually has a local structure. That is also a tensor logic equation. Now we are saying, how do you choose between the convnet and then MLP, right? Very good question. And now there's a range of things you can do. You can actually these days start out with a very general structure because, I mean, GPUs and and and large, you know, server farms are an amazing amount of power for something like this. Right? So you can almost, I would say, brute force that search provided you have the data. I'm not actually recommending you to do that, right? You can also, however, and more interestingly, you can, and this is actually 1 of the key benefits of tensor logic is that you can write down what you believe are properties of the say, right now what happens in in in in, know, when you for example program a network in Python is like, you have to commit, you say like, here's the structure, and now the, you know, the the learning of the only thing that happens is the learning of the weights. Intensive logic you don't have to do that. You can you can just you can set up 1 of these very general structures and then you say, let me give you a bunch of equations that are things that I believe to be true about the structure but do not completely determine it. And those just work like priors, and indeed like soft priors, right? Those can and then again you can turn up the temperature on this or down, and say like, you got to obey this equation, and that 1, you know, sure you can overwrite. And then and then this, in my experience, this is actually what is important, is that the gradient descent, instead of starting from a tabular rasa, have this kind of soft knowledge. And then most importantly like you, the developer, you the AI researcher, you you get to this is really the essence is that it's not this you every every, you know, deep learning researcher or data scientist know this. It's like you don't do the same priority and then push the button and hope for the best, right? It's like there's an iterative loop of you set up the structure and then you learn, you get the results, and then you refine the structure. And what this does is it makes that more efficient much more because you just have to in your interpreter you write 1 more equation or you modify an existing equation, and also the entire stack is is of what you learn is much more interpretable than it was before. It's actually in some ways 1 of the most important properties of tensor logic, is that you can understand what's going on much better than you could in 2 ways.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.