Petar Velichkovich: evaluation

Transcript context

…Geometric deep learning is powerful, but it assumes all transformations are invertible. What happens when computation destroys information? Strictly speaking, I wasn't planning to talk about any of that stuff at the time. It was very much work in progress and just kind of trapped in my head as a collection of possible ideas, but not something I remotely knew how to execute on. Right? But I was very passionate about algorithmic reasoning at the time as well. I still am. And I still believe that building machine learning models that are capable to align to classical computation is going to be really, really important to address the shortcomings that are not so easily plugged by just gathering a better data set. Fundamentally, some of these things are likely to be restricted to not be able to easily generalize outside of the distribution you've trained them on. And especially for reasoning problems, that is the case. When you think about all of the big scientific advances that were done with large language models, for example, up to this date, I would argue most of the ones I'm personally familiar with are a result of a careful combination of a large language model and an algorithmic procedure in the background, which actually makes sure to give it robustness properties. So think about things like fund search, like alpha code, like alpha geometry. All of these systems have discovered new knowledge in computer science, in competitive programming, in, in even, you know, geometry problems at the IMO. But in all cases, you've hooked up a language model to either a genetic algorithm or some clustering mechanism or a tier improver, which all have very nice correctness guarantees, which then, if you can run this model for sufficiently many times to kind of correct itself using the algorithm, you can end up with really nice solutions. The problem with geometric deep learning is that, as I said, it talks about symmetries. So permutations or circular shifts, those are generally things that have very specific and rigid behaviors. And typically, 1 of the things we assume about symmetries is that they are invertible. So basically, whenever I permute nodes, I can always permute them back. I haven't lost any information. And usually with images, when we do shifts, we actually pad the image with zeros to make sure that no image data data is lost and things like that. So basically, it always assumes that it's still the same input. We haven't lost any information. Now, is this a problem for me, is really interested in aligning models to classical algorithmic computation? Well, as any computer scientist will know, many programs you write will delete some of the data or destroy some of the data. So that is no longer a symmetry. You cannot invert it. Maybe 1 simple example is other than just the naive example of, I'm going to take a list and delete half of its elements for no reason. Let's start with pathfinding. So we talk a lot about algorithms like Dijkstra or Bellman Ford inside a computer science curriculum. going to take a list and delete half of its elements for no reason. Let's start with pathfinding. So we talk a lot about algorithms like Dijkstra or Bellman Ford inside a computer science curriculum. In short, those are algorithms that, starting with a directed weighted graph, predict what are the shortest paths lengths inside that graph. Right? Now, the thing is there are many, many different graphs with different weights that are going to have exactly the same shortest paths and potentially even the same shortest path lengths. However, those graphs are different. And once you've applied the transformations of Dijkstra's algorithm or Bellman Ford algorithm, you'll have lost the information that is contained about the graph in the final output of that algorithm, right? Because many different graphs will be compressed to exactly the same output, right? So this is not an operation I can describe using asymmetry. And then this journey gradually like, took me a while to realize how we can be formal about this, how can we try to put some theory on this, and even now down the line, how can we build some practical models using this. And I was fortunate enough to to start chatting with Andrew, who is my colleague at DeepMind, who has a category theory background. And he's been thinking himself about some of these problems in the past. So it was a great match because together with him, it was a long way. But we managed to gradually relax the constraints of what a group is giving us. We first looked at removing the invertibility part, which led us to monoids. And then we derived some interesting theory and asynchrony invariance in models using monoids. And now we're also looking into removing the second constraint of groups, which is the requirement that every single computation must compose with every other piece of computation. As you might also know in computer science, you cannot always do that. You must make the output of your first function match the input type of the second 1. Otherwise, they can't compose. So this now leads us to categories. And well, that's what led us now to categorical deep learning.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.