Andrew Dudzik: evaluation

Transcript context

…So what is the syntax, for example, of of the action of a group? It's really you kind of say, well, I I wanna just think I have like 1 kind of thing, and then each group element takes that thing and makes it sort of sends it back to itself. So I have sort of a single type, and then each group action does something to that type. So I might have some points in the plane and my group might be rotations and reflections that move those points around, but I'm still in the plane. But it turns out that this 1 sorted syntax isn't enough to to even capture basic type constructors in in computer science. So for example, lists, you cannot deal with the syntax of lists using just a single sort. You need a multi sorted syntax. So the the way that you can do this is to say, well, think of we have sort of 0 tuples, 1 tuples, 2 tuples, etcetera. And then given k tuple, I might be able to make some other kind of tuple, an l tuple, by taking elements of my k tuple and making them into some lists, l lists. Right? So that's that's a syntax in the same way that group elements were a syntax. And it's it does have compositionality. If I have a way of packing things in from a tuple into a bunch of lists, then I can pack those lists into other tuples of lists. But it's clear that this is first of all many sorted and then also differs from the group case because all of this is highly non invertible. Right? There's you you can't sort of by packing things more and more and more into lists eventually undo the lists. Right? You just get more lists. But we basically construct a model for the syntax. And in the case of groups, a model in sets means a set acted on by the group. While, for example, a model in vector spaces would mean a vector representation of the group. Whereas in the case of the syntax for lists, you just get what I was calling before foldable types or monoids. Things that are able to perform these syntactic operations in the expected way. Like, there's just something, where the mathematical reasoning just works better if you expand your universe of objects a little bit. Even if you only care about the original objects. So this is a lesson that mathematicians have learned many times and it's sort of why a lot of people will prefer to work on the semantic side, at least in the first instance. It turns out that there's something very very basic in mathematics that we all learned in elementary school, that has sort of been overlooked in the design of GNNs, and that's the notion of a carry. So what exactly is a carry? Well, suppose that I am able to implement a device, a number wheel, that can do arithmetic modulo 10. So 0 through 9 on it. And now I want to build a kind of composite wheel that can do arithmetic modulo 100. So what do I need to do? I need sort of this little mechanism that when it when the wheel goes from 9 to 0, it turns the next wheel by 1. d a kind of composite wheel that can do arithmetic modulo 100. So what do I need to do? I need sort of this little mechanism that when it when the wheel goes from 9 to 0, it turns the next wheel by 1. And this is very simple but it's extremely at odds with the way that GNNs have been conceived of in the past because in the past you generally, send the whole state but there's no information in the state. The information is only in the change of the state. But it's even worse than that. Even if you sent the change in this state, that's not enough information because if I went from 9 to 0, is it because I added 1? Is it because I added 11? Is it because I subtracted 9? It turns out that it's quite, it's quite subtle to, get this kind of thing to work in the presence of gradient descent. So it somehow is a very fundamental aspect of how we assemble more complicated computational operations from simpler ones. I mean, 1 of the first things that you do if you're describing a CPU is you have to describe an adder. This is already something that we're struggling to do in in GNN terms. It turns out that this behavior is easy to get when you do discrete mathematics and very complicated to get when you do continuous mathematics. You can easily give this number wheel example. Everybody understands it because they know how to do addition. But getting it to happen in a way such that everything is continuous turns out to be really interesting. The simplest examples of this phenomenon, don't occur until you're dealing with three-dimensional manifolds. So you would need to be thinking about things in 4 dimensional space. And the simplest example that we know of is the so called hop vibration. This is a situation where, you can decompose a three-dimensional sphere, so that's a sphere in 4 dimensions. You can project it onto a 2 dimensional sphere so that all of the pre images are 1 dimensional spheres or circles. And the three-dimensional sphere is very different from the product of the 1 and 2 dimensional spheres. Just the same way that z mod 100 is very different from the product of z mod 10 with z mod 10. And so this is something that I'm personally very excited about right now coming out of this asynchrony work is, are there ways to exploit this type of geometric subtlety to create the phenomenon of carrying and actually properly model this aspect of algorithmic reasoning and and start to build actual CPUs in neural networks. So their claim is quite straightforward at the end of the day. Deep learning has 2 languages, constraints and implementation. And we lack a single framework that cleanly links them together. Categorical deep learning produces the bridge. Right, using a universal algebra in a 2 category of parametric maps. It recovers geometric deep learning as a special case while naturally expressing things like recursion, weight tying, and non invertible computation. Now if you want the formal story, go and read their paper. The link is in the description, especially the sections on para, weight tying and recovering geometric deep learning. Cool. Thanks for watching.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.