Andrew Gordon Wilson: evaluation

Transcript context

…Right. Really, what we care about is the properties of this sort of induced distribution over functions rather than just how many parameters the model happens to have. And so you can have a distribution over functions which is very flexible. It can represent many different solutions to a given problem, but it can also have very strong preferences for certain types of solutions over others. And strong preferences doesn't mean saying that certain things are impossible necessarily. They can just have Epsilon probability. And I think that that is really meaningfully different than saying, okay, we're we're gonna have a hard constraint and we're not gonna represent those solutions. A, because it's not an honest representative of our representation of our beliefs to to have those hard constraints, and b, because we see in practice that when we do have these expressive models with simplicity biases, they're much more adaptive. They're much more automatic. So when you have a small data set, it sort of does the right thing. When you have a large data set, it also does the right thing. And so you don't need as much human intervention. And arguably, that's the definition of what really machine learning is trying to achieve. It's trying to build an intelligent system that doesn't require manual intervention. And so I think this is an important principle towards that goal. You know, you you had a paper kind of removing some of the mysteries of deep learning. But I think it's it's still fair to say it's a bit mysterious where the simplicity bias comes from at scale, isn't it?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.