Dwarkesh Patel: uncertainty

Transcript context

…Exactly, yeah, or like someone worked on it, it was awesome, it got far, but, another excited Anthropic employee then iterated on the memory scaffold and was able to very quickly improve on it. So that's one. I do also just wonder if models are idiot savants. The best analogy might be to Kim Peek. So Kim Peek, was born without a corpus callosum, if I recall correctly. Each hemisphere of his brain operated quite independently. So, he'd open a book, there'd be two pages visible, each eye would read one of the pages. And he had a perfect encyclopedic memory of everything he'd ever read. But at the same time, he had other debilitations; functioning socially, these sorts of things. And it's just kind of amazing how good LLMs are at very niche topics, but can totally fail at other ones still. I really wanna double-click on this thing of why there's this trade-off between memorization. Why does cutting it off... apparently it's connected to this debilitation, but why can't… Wiki text is like five megabytes of information. The human brain can store much more, so why does the human brain just not want us to memorize these kinds of things, and is actively pruning, and… yeah, I don't know. But we don't have to do it right now. We'll do a separate episode. Yeah, just one thing I'll say on that is, there is another case study of someone with a perfect memory, so they never forgot anything. But their memory was too debilitating. It'd be, like, your context window for the transformer is trillions of tokens. And then you spend all your time attending to past things, and are too trapped in the details to extract any meaningful generalizable insights from it.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.