Lex Fridman: belief

Transcript context

…Okay, so now instead of training a system to encode the image and then training it to reconstruct the full image from a corrupted version, you take the full image, you take the corrupted or transformed version, you run them both through encoders, which in general, are identical, but not necessarily. And then you train a predictor on top of those encoders to predict the representation of the full input from the representation of the corrupted one. So joint embedding, because you’re taking the full input and the corrupted version or transformed version, run them both through encoders, you get a joint embedding, and then you’re saying, can I predict the representation of the full one from the representation of the corrupted one? And I call this a JEPA, so that means joint embedding predictive architecture because this joint embedding and there is this predictor that predicts the representation of the good guy from the bad guy. And the big question is how do you train something like this? And until five years ago or six years ago, we didn’t have particularly good answers for how you train those things except for one, called contrastive learning, where the idea of contrastive learning is you take a pair of images that are, again, an image and a corrupted version or degraded version somehow or transformed version of the original one, and you train the predicted representation to be the same as that. If you only do this, this system collapses. It basically completely ignores the input and produces representations that are constant. So the contrastive methods avoid this, and those things have been around since the early ’90s, I had a paper on this in 1993, is you also show pairs of images that you know are different, and then you push away the representations from each other. So you say, not only do representations of things that we know are the same should be the same or should be similar, but representation of things that we know are different should be different. And that prevents the collapse, but it has some limitation. And there’s a whole bunch of techniques that have appeared over the last six, seven years that can revive this type of method, some of them from FAIR, some of them from Google and other places, but there are limitations to those contrastive methods. What has changed in the last three, four years is now we have methods that are non-contrastive. So they don’t require those negative contrastive samples of images that we know are different. You turn them on you with images that are different versions or different views of the same thing, and you rely on some other tricks to prevent the system from collapsing. And we have half a dozen different methods for this now. So what is the fundamental difference between joint embedding architectures and LLMs? Can JEPA take us to AGI? Whether we should say that you don’t like the term AGI, and we’ll probably argue I think every single time I’ve talked to you, we’ve argued about the G in AGI. Yes.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.