Dwarkesh Patel: evaluation

Transcript context

…Sucking supervision through a straw, as Karpathy says? Exactly. Of course people are working on many different techniques, but fundamentally there’s this big constraint in the way we train AIs. With code, you can containerize a given level of progress in a repository and then spin out hundreds of parallel containers and say, “Try to implement this feature,” and it’s totally deterministic. Because it’s deterministic, you can solve the credit assignment problem because you know that whatever caused this rollout to succeed and this one to fail, the diff is the thing that worked. If you have situations that are starting off at different starting points, this credit assignment problem becomes much harder to solve. Most things in the real world are very hard to containerize in the same way. Coding and math are exceptions to this rule. But if you’re trying to figure out how to build a new business that succeeds, or how to go trade in the markets for a day and make money, the fact that you have to interact with the real world and things change day after day means that you can’t keep replaying and grinding and farming the simulator. Math, of course, is the exception, and I feel like this is an important driver of progress in this domain and also in coding. It’s not just verifiability; it has to be grindable. The third reason people point out that AI is making fast progress is they focus a lot on Lean and formalization. Again, I have literally no idea what’s going on in the labs. I feel like Lean just doesn’t matter that much for the current level of progress in AI. Why is AI able to disprove the conjecture about the unit distance problem? They released the chain of thought, or at least a rewrite of the chain of thought. It didn’t have any Lean in it. I think the process-based supervision that Lean provides, where you know each step is correct, seems less relevant than just having this grindable outcome that is verifiable. It’s an interesting point about grindability mattering more. Naively you might think Lean provides something unique for math because you’re able to see if it can prove it. You have old-school software that can tell you yes or no, and you use that as your VR. What would corroborate your point is the initial attempts. Again, I’ll circle back to the IMO. Initially, DeepMind basically does that. Everything is in Lean, and then the next year it’s all in natural language. So to your point, it’s not needed. I do think there’s a yet-to-be-explored benefit of that formalization domain, which is that at the moment you still need a human reviewing that counterexample to the unit distance conjecture to say, “Looks good.” That provides a certain bound on how endlessly explorable things are. If you consider AlphaGo or AlphaZero-style systems, they’re off in their own universe playing a bunch of Go and exploring themselves, potentially going off the rails of what any human needs to look at, but they still have this automated verifiable reward. It’s not just that you can do RL on that. It’s also that you basically never have to check in, and you can just pour compute at them exploring the universe of Go. What stands to be interesting—maybe this won’t pan out, but the jury should still be out on whether it’ll yield anything—is that with Lean, you could imagine having a basically endlessly running program that’s constantly trying to extend Mathlib. Mathlib is this GitHub repository that’s basically all of math written in code. It’s very far from all of math, but they want it to be all of math. It’s written in code where you can ask, “Is this proof correct?” It’s very labor-intensive to write these proofs. There’s a whole subcommunity around it. But you could imagine having an AI where you say, “Simply try to extend Mathlib.” Maybe it’s a fork of it so that it doesn’t have trash in it, because people have a certain taste for what they want to be in there. So you have your fork of the pure AI Mathlib, and it just goes and it doesn’t stop. It doesn’t need anybody to check in on it. It could just keep going. It might come up with its own conjectures. It might come up with its own theories and different definitions. Maybe many of them are useless, but it just has this infinite tree that it can grow out. That’s a very unique thing that math has that nothing else has, where you could press go and just pour compute at it, look away for ten years, and then come back and say, “What do you have?” There’s going to be something. Then there’s a question: is it useful or not? How do you suss that out? That’s just an interesting thing to be able to do. It would be very surprising if that didn’t yield some sort of interesting mathematical insight from it. There are two different ways that Lean is important in this story. The first one is how you could let go, not even check in, and progress will be made. You can do that with Go.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.