Dwarkesh Patel: belief

Transcript context

…That’s one of them. I think there are a lot of deeper questions that one could tackle. For example, let’s say you have an idea on how to improve a scaling-law compute multiplier. The outcome isn’t necessarily “achieve the best Go bot ever”. The outcome might just be, “Can I predict what the win rate of my Go bot will be?” Or, “Can I predict the scaling-law plots that emerge from my idea?” But then you can verify that you haven’t reward-hacked anything by using a very verifiable game like Go on the outer loop. I think there are a couple of interesting follow-on questions. There are questions on the inner loop and the outer loop. On the inner loop, there’s a question of how locally verifiable any modification you might make is. That is to say, would you know whether some idea you try out is actually an improvement or a degradation? Would you know if something isn’t working as a result of a bug, or is it the result of the idea itself being wrong? Ilya was talking about how one of the things he thinks makes him a good researcher is that he has a strong belief in what the correct idea is. He’s able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea, based on his high-level belief that “this idea should work, so therefore there has to be a bug”, versus the other way around. Why don’t we start with that question? How locally verifiable are things which are good ideas? As in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. How do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don’t have an answer to. Ultimately, to play a very strong Go bot, you probably did need to discover deep learning. Having a challenging game that cannot be cheated easily on the outer loop could be used as an outer-loop signal for something like discovering the principles of deep learning. Now, of course, to make it tractable—and this is where research taste really matters—you have to come up with ways to initialize your problem so that you don’t try to solve a very intractable problem. Maybe you can leverage LLMs as a universal grammar in the middle to give you some sort of local feedback. The fact that LLMs are a universal grammar means that they can move at almost any level of the stack. They can think very locally, as well as step back and think in very broad steps. I think that’s where a lot of the lateral thinking ability of humans comes from: knowing when the track you’re pursuing or the objective you’re pursuing is not right, and you should be asking a different question.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.