Lex Fridman: belief

Transcript context

…I mean, there’s 16,000 hours of wake time of a 4-year-old and tend to do 15 bites going through vision, just vision, there is a similar bandwidth of touch and a little less through audio. And then text, language doesn’t come in until a year in life. And by the time you are nine years old, you’ve learned about gravity, you know about inertia, you know about gravity, the stability, you know about the distinction between animate and inanimate objects. You know by 18 months, you know about why people want to do things and you help them if they can’t. I mean, there’s a lot of things that you learn mostly by observation, really not even through interaction. In the first few months of life, babies don’t really have any influence on the world, they can only observe. And you accumulate a gigantic amount of knowledge just from that. So that’s what we’re missing from current AI systems. I think in one of your slides, you have this nice plot that is one of the ways you show that LLMs are limited. I wonder if you could talk about hallucinations from your perspectives, the why hallucinations happen from large language models and to what degree is that a fundamental flaw of large language models? Right, so because of the autoregressive prediction, every time an produces a token or a word, there is some level of probability for that word to take you out of the set of reasonable answers. And if you assume, which is a very strong assumption, that the probability of such error is that those errors are independent across a sequence of tokens being produced. What that means is that every time you produce a token, the probability that you stay within the set of correct answer decreases and it decreases exponentially.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.