Jeremy Howard: observation

Transcript context

…But the thing is, I want to contrast this with other comments you've made about creativity. So I think knowledge is about constraints. And I think creativity is the evolution of knowledge, respecting those constraints. Therefore, AI is not creative. And you've said the same thing. You've said AI isn't creative. So like, on the 1 hand, how can you say that they know and not think that they can be created? I mean, I don't think I've used that exact expression. You know, I know I've actually I remember chatting with Peter Norvig on camera, and both of us said, well, actually, they kind of are creative, like, we just gotta be a bit careful about our choices of words, I guess. So, you know, Piotr Wozniak, who's a guy I really really respect, who kind of rediscovered spaced repetition learning, built the SuperMemo system, and is the modern day guru of memory. The entire reason he's based his life around remembering things is because he believes that creativity comes from having a lot of stuff remembered, which is to say putting together stuff you've remembered in interesting ways is a great way to be creative. LLMs are actually quite good at that, but there's a kind of creativity they're not at all good at, which is, you know, moving outside the distribution. So which I think is where you're heading with your question. But I'm just kind of I'm framing it this way to say, you have to be so nuanced about this stuff because if you say like they're not creative, it gives you the can give you the wrong idea because they can do very creative seeming things. But if it's like, well, can they really extrapolate outside the training distribution? The answer is no, they can't. But the training distribution is so big, and the number of ways to interpolate between them is so vast, we don't really know yet what the limitations of that is. But I see it every day, you know, because I my my work is R and D. I'm constantly on the edge of and outside the training data. I'm doing things that haven't been done before. And there's this weird thing, I don't know if you've ever seen it before, I see it but I see it multiple times every day, where the LM goes from being incredibly clever to like worse than stupid, like like not understanding the most basic fundamental premises about how the world works. Yeah. Yeah.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.