Jeff Bezos: evaluation

Transcript context

…And it could be a few tricks and hacks here and there that open doors to hold entire new possibilities. We do know that humans are doing something different from these models, in part because we’re so power efficient. The human brain does remarkable things and it does it on about 20 watts of power. And the AI techniques we use today use many kilowatts of power to do equivalent tasks. So there’s something interesting about the way the human brain does this. And also we don’t need as much data. So self-driving cars, they have to drive billions and billions of miles to try to learn how to drive. And your average 16-year-old figures it out with many fewer miles. So there are still some tricks, I think, that we have yet to learn. I don’t think we’ve learned the last trick. I don’t think it’s just a question of scaling things up. But what’s interesting is that just scaling things up, and I put just in quotes because it’s actually hard to scale things up, but just scaling things up also appears to pay huge dividends. Yeah. And there’s some more nuanced aspect about human beings that’s interesting if it’s able to accomplish like being truly original and novel. Large language models, being able to come up with some truly new ideas. That’s one. And the other one is truth. It seems that large language models are very good at sounding like they’re saying a true thing, but they don’t require or often have a grounding in a mathematical truth, basically is a very good bullshitter. So if there’s not enough data in the training data about a particular topic, it’s just going to concoct accurate sounding narratives, which is a very fascinating problem to try to solve, how do you get language models to infer what is true or not to introspect?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.