Yann LeCun

The two techniques we've developed at FAIR, DINO and I-JEPA work really well for that.

The two techniques we've developed at FAIR, DINO and I-JEPA work really well for that.

the reason why we have multilingual translation system, things to do, content moderation on Meta, for example, on Facebook that are multilingual, that understand whether piece of text is hate speech or not, or something is due to their progress using self supervised running for NLP, combining this with transformer architectures

the reason why we have multilingual translation system, things to do, content moderation on Meta, for example, on Facebook that are multilingual, that understand whether piece of text is hate speech or not, or something is due to their progress using self supervised running for NLP, combining this with transformer architectures

We had similar success in speech recognition, a system called Wav2Vec, which is also a joint embedding architecture by the way, trained with contrastive learning. And that system also can produce speech recognition systems that are multilingual with mostly unlabeled data and only need a few minutes of labeled data to actually do speech recognition.

It started working, we abandoned this idea of predicting every pixel and basically just doing the joint embedding and predicting in representation space. That works.

If you had such a model, you could use it to produce good answers. The way you would do is produce the prompt and then search through the space of possible answers for one that minimizes that number.

there is literally millions of downloads of LLaMA 2 and thousands of people who have provided ideas about how to make it better.

There's similar work at DeepMind also taking place, and also at UC Berkeley on world models and video. A lot of people are working on this. I think a lot of good ideas are appearing.

There's similar work at DeepMind also taking place, and also at UC Berkeley on world models and video. A lot of people are working on this. I think a lot of good ideas are appearing.

There's similar work at DeepMind also taking place, and also at UC Berkeley on world models and video. A lot of people are working on this. I think a lot of good ideas are appearing.

There's similar work at DeepMind also taking place, and also at UC Berkeley on world models and video. A lot of people are working on this. I think a lot of good ideas are appearing.

you see this along history, like any technological revolution or cultural phenomenon was always accompanied by groups or reaction in the media that basically attributed all the problems, the current problems of society to that particular change, right?

Electricity was going to kill everyone at some point. The train was going to be a horrible thing because you can't breathe past 50 kilometers an hour. And so there's a wonderful website called a Pessimists Archive, right?

they've been not building their own robots but using commercial robots. And you can tell the robot dog like go to the fridge and they can actually open the fridge and they can probably pick up a can in the fridge and stuff like that and bring it to you. So it can navigate, it can grab objects as long as it's been trained to recognize them, which vision systems work pretty well nowadays.

I see the danger of this concentration of power through proprietary AI systems as a much bigger danger than everything else.

I believe that people are fundamentally good and so if AI, especially open source AI can make them smarter, it just empowers the goodness in humans.

You've said that autoregressive LLMs are not the way we're going to make progress towards superhuman intelligence.

LLMs can do none of those, or they can only do them in a very primitive way. And they don't really understand the physical world, they don't really have persistent memory, they can't really reason and they certainly can't plan.

they don't really understand the physical world, they don't really have persistent memory, they can't really reason and they certainly can't plan. And so if you expect the system to become intelligent just without having the possibility of doing those things, you're making a mistake.

That is not to say that autoregressive LLMs are not useful, they're certainly useful. That they're not interesting, that we can't build a whole ecosystem of applications around them, of course we can.

those LLMs are trained on enormous amounts of text. Basically the entirety of all publicly available text on the internet, right? That's typically on the order of 10 to the 13 tokens. Each token is typically two bytes. So that's two 10 to the 13 bytes as training data. It would take you or me 170,000 years to just read through this at eight hours a day.

a 4-year-old has been awake for 16,000 hours in his or her life, and the amount of information that has reached the visual cortex of that child in four years is about 10 to 15 bytes. And you can compute this by estimating that the optical nerve carry about 20 megabytes per second, roughly.

most of what we learn and most of our knowledge is through our observation and interaction with the real world, not through language. Everything that we learn in the first few years of life, and certainly everything that animals learn has nothing to do with language.

I'm clearly in the camp that yes, intelligence cannot appear without some grounding in some reality.

we have a number of ways to train vision systems, either supervised, unsupervised, self-supervised, all kinds of different ways. That will turn any image into a high level representation. Basically, a list of tokens that are really similar to the kind of tokens that a typical LLM takes as an input. And then you just feed that to the LLM in addition to the text, and you just expect the LLM during training to kind of be able to use those representations to help make decisions.

there are LLMs that have some vision extension. But they're basically hacks in the sense that those things are not like trained to handle, to really understand the world. They're not trained with video, for example. They don't really understand intuitive physics, at least not at the moment.

So you don't think there's something special to you about intuitive physics, about sort of common sense reasoning about the physical space, about physical reality? That to you is a giant leap that LLMs are just not able to do?

We're not gonna be able to do this with the type of LLMs that we are working with today.

the way LLMs are trained is that you take a piece of text, you remove some of the words in that text, you mask them, you replace them by black markers, and you train a gigantic neural net to predict the words that are missing. And if you build this neural net in a particular way so that it can only look at words that are to the left of the one it's trying to predict, then what you have is a system that basically is trying to predict the next word in a text, right?

It can never predict the next word exactly. And so what it's gonna do is produce a probability distribution of all the possible words in the dictionary. In fact, it doesn't predict words, it predicts tokens that are kind of subword units. And so it's easy to handle the uncertainty in the prediction there because there's only a finite number of possible words in the dictionary, and you can just compute a distribution over them. Then what the system does is that it picks a word from that distribution. Of course, there's a higher chance of picking words that have a higher probability within that distribution. So you sample from that distribution to actually produce a word, and then you shift that word into the input. And so that allows the system now to predict the second word, right? And once you do this, you shift it into the input, et cetera. That's called autoregressive prediction, which is why those LLMs should be called autoregressive LLMs, but we just call them at LLMs.

And there is a difference between this kind of process and a process by which before producing a word, when you talk. When you and I talk, you and I are bilinguals. We think about what we're gonna say, and it's relatively independent of the language in which we're gonna say. When we talk about like, I don't know, let's say a mathematical concept or something.

the idea of JEPA is really about. Learn abstract representation in a self supervised manner. And you can do it hierarchically as well. So that I think is an essential component of an intelligent system.

And in language, we can get away without doing this because language is already to some level abstract and already has eliminated a lot of information that is not predictable.

those self supervised algorithms that learn by prediction, even in representation space, they learn more concept if the input data you feed them is more redundant. The more redundancy there is in the data, the more they're able to capture some internal structure of it.

there is way more redundancy in the structure in perceptual inputs, sensory input like vision, than there is in text, which is not nearly as redundant.

We're basically cheating. We are using language as a crutch to help the deficiencies of our vision systems to kind of learn good representations from images and video.

But we're not gonna get to the level of even the intelligence or level of understanding of the world of a cat or a dog which doesn't have language.

How do we get machines to learn that before we combine that with language?

In fact, the techniques we're using are non-contrastive. So not only is the architecture non-generative, the learning procedures we're using are non-contrastive.

The two techniques we've developed at FAIR, DINO and I-JEPA work really well for that.

So there's several scenarios. One scenario is you take an image, you corrupt it by changing the cropping, for example,

in the case of I-JEPA, you don't need to do any of this, you just mask some parts of it, right? You just basically remove some regions like a big block, essentially.

It doesn't need to know that it's an image, for example, because the only thing it needs to know is how to do this masking. Whereas with DINO, you need to know it's an image because you need to do things like geometry transformation and blurring and things like that that are really image specific.

A more recent version of this that we have is called V-JEPA. So it's basically the same idea as I-JEPA except it's applied to video. So now you take a whole video and you mask a whole chunk of it.

It's the first system that we have that learns good representations of video so that when you feed those representations to a supervised classifier head, it can tell you what action is taking place in the video with pretty good accuracy.

preliminary result that seem to indicate that the representation allows our system to tell whether the video is physically possible or completely impossible because some object disappeared or an object suddenly jumped from one location to another or changed shape or something.

Possibly. And this is gonna take a while before we get to that point. And there are systems already, robotic systems, that are based on this idea.

the reason why we have multilingual translation system, things to do, content moderation on Meta, for example, on Facebook that are multilingual, that understand whether piece of text is hate speech or not, or something is due to their progress using self supervised running for NLP, combining this with transformer architectures

We had similar success in speech recognition, a system called Wav2Vec, which is also a joint embedding architecture by the way, trained with contrastive learning. And that system also can produce speech recognition systems that are multilingual with mostly unlabeled data and only need a few minutes of labeled data to actually do speech recognition.

We have systems now based on those combination of ideas that can do real time translation of hundreds of languages into each other, speech to speech. - Speech to speech, even including, which is fascinating, languages that don't have written forms- - That's right. - They're spoken only. - That's right. We don't go through text, it goes directly from speech to speech using an internal representation of kinda speech units that are discrete.

there's ample evidence that we're not gonna be able to learn good representations of the real world using generative model.

If you're really interested in human level AI, abandon the idea of generative AI.

It started working, we abandoned this idea of predicting every pixel and basically just doing the joint embedding and predicting in representation space. That works.

I mean, that's the 16,000 hours of wake time of a 4-year-old. And tend to do 15 bytes, going through vision. Just vision, right? There is a similar bandwidth of touch and a little less through audio. And then text doesn't... Language doesn't come in until like a year in life.

In the first few months of life, babies don't really have any influence on the world. They can only observe, right? And you accumulate like a gigantic amount of knowledge just from that.

So there's a strong, like you said, assumption there that if there's a non-zero probability of making a mistake, which there appears to be, then there's going to be a kind of drift. - Yeah. And that drift is exponential. It's like errors accumulate, right? So the probability that an answer would be nonsensical increases exponentially with the number of tokens.

the probability that an answer would be nonsensical increases exponentially with the number of tokens

it's basically a struggle against the curse of dimensionality

the way you can correct for this is that you fine tune the system by having it produce answers for all kinds of questions that people might come up with

within the set of all possible prompts, the proportion of prompts that have been used for training is absolutely tiny

the system will behave properly on the prompts that it's been either trained, pre-trained or fine tuned. But then there is an entire space of things that it cannot possibly have been trained on because it's just the number is gigantic

you put essentially a random sequence of characters in a prompt and that's enough to kind of throw the system into a mode where it's gonna answer something completely different than it would have answered without this

you write a sentence in English or you ask a question in English and it produces a perfectly fine answer. And then you just substitute a few words by the same word in another language, and all of a sudden the answer is complete nonsense

I think there's going to be a lot of systems over the next few years that are going to have this capability, but the blueprint of those systems will be extremely different from autoregressive LLMs.

the blueprint of those systems will be extremely different from autoregressive LLMs.

So system one is the type of task that you can accomplish without like deliberately consciously think about how you do them. You just do them.

And so this type of deliberate planning, which uses your internal world model, that's system two, this is what LLMs currently cannot do.

And it's not going to be autoregressive prediction of tokens, it's going to be more something akin to inference of latent variables in what used to be called probabilistic models or graphical models and things of that type.

The prompt is like observed variables. And what the model does is that it's basically a measure of... It can measure to what extent an answer is a good answer for a prompt.

If you had such a model, you could use it to produce good answers. The way you would do is produce the prompt and then search through the space of possible answers for one that minimizes that number.

Meta's ability to derive revenue from this technology is not impaired by the distribution of base models in open source.

there is literally millions of downloads of LLaMA 2 and thousands of people who have provided ideas about how to make it better.

this clearly accelerates progress to make the system available to sort of a wide community of people.

a big company cannot afford to offend too many people. So they're going to make sure that whatever product they put out is "safe," whatever that means.

you cannot have a system that is unbiased and is perceived as unbiased by everyone.

I don't think the issue has to do with the political leaning of the people designing those systems.

It has to do with the acceptability or political leanings of their customer base or audience, right?

Basically the technology enables humans to human more effectively.

My bet is that those systems are gonna be JEPA-like, they're not gonna be generative models.

There's similar work at DeepMind also taking place, and also at UC Berkeley on world models and video. A lot of people are working on this. I think a lot of good ideas are appearing.

I see a path towards potentially human level intelligence with systems that can understand the world, remember, plan, reason.

certainly scale is necessary but not sufficient.

we're still far in terms of compute power from what we would need to match the compute power of the human brain. This may occur in the next couple decades, but we're still some ways away. And certainly in terms of power efficiency, we're really far.

designing those guardrails so that the system behaves properly is not gonna be a simple issue for which there is a silver bullet, for which you have a mathematical proof that the system can be safe. It's gonna be very progressive, iterative design system where we put those guardrails in such a way that the system behave properly.

And there is several arguments that make people scared of this, which I think are completely false as well. So one of them is in nature, it seems to be that the more intelligent species are the ones that end up dominating the other. And even extinguishing the others sometimes by design, sometimes just by mistake.

And to you, there's not significant incentive for humans to encode that into the AI systems.

objective driven AI, so systems that derive their answers by optimization of an objective means they have to optimize this objective, and that objective can include guardrails.

there is a natural fear of new technology and the impact it can have on society. And people have kind of instinctive reaction to the world they know being threatened by major transformations that are either cultural phenomena or technological revolutions.

you see this along history, like any technological revolution or cultural phenomenon was always accompanied by groups or reaction in the media that basically attributed all the problems, the current problems of society to that particular change, right?

Electricity was going to kill everyone at some point. The train was going to be a horrible thing because you can't breathe past 50 kilometers an hour. And so there's a wonderful website called a Pessimists Archive, right?

Wonderful examples of jazz or comic books being blamed for unemployment or young people not wanting to work anymore and things like that, right?

that's exactly why we need open source platforms.

they've been not building their own robots but using commercial robots. And you can tell the robot dog like go to the fridge and they can actually open the fridge and they can probably pick up a can in the fridge and stuff like that and bring it to you. So it can navigate, it can grab objects as long as it's been trained to recognize them, which vision systems work pretty well nowadays.

But it's not like a completely general robot that would be sophisticated enough to do things like clearing up the dinner table.

we've been like kinda working on this idea of self supervised learning from video for 10 years. And only made significant progress in the last two or three.

there's a lot of interesting breakthroughs that can happen without having access to a lot of compute. So if you're interested in doing a PhD in this kind of stuff, there's a lot of possibilities still to do innovative work.

If the world the system evolves in is not the physical world, but is the world of let's say the internet or some sort of world of where an action consists in doing a search in a search engine or interrogating a database, or running a simulation or calling a calculator or solving a differential equation, how do you get a system to actually plan a sequence of actions to give the solution to a problem?

We can do like two level hierarchical planning when we design the two levels. So for example, you have like a dog legged robot, right? You want it to go from the living room to the kitchen. You can plan a path that avoids the obstacle. And then you can send this to a lower level planner that figures out how to move the legs to kind of follow that trajectories, right? So that works, but that two level planning is designed by hand, right? We specify what the proper levels of abstraction, the representation at each level of abstraction have to be. How do you learn this? How do you learn that hierarchical representation of action plans, right? With com nets and deep learning, we can train the system to learn hierarchical representations of percepts. What is the equivalent when what you're trying to represent are action plans?