Ilya Sutskever: belief

Transcript context

…If we’re back in the era of research, stepping back, what is the part of the recipe that we need to think most about? When you say value function, people are already trying the current recipe, but then having LLM-as-a-Judge and so forth. You could say that’s a value function, but it sounds like you have something much more fundamental in mind. Should we even rethink pre-training at all and not just add more steps to the end of that process? The discussion about value function, I think it was interesting. I want to emphasize that I think the value function is something that’s going to make RL more efficient, and I think that makes a difference. But I think anything you can do with a value function, you can do without, just more slowly. The thing which I think is the most fundamental is that these models somehow just generalize dramatically worse than people. It’s super obvious. That seems like a very fundamental thing. So this is the crux: generalization. There are two sub-questions. There’s one which is about sample efficiency: why should it take so much more data for these models to learn than humans? There’s a second question. Even separate from the amount of data it takes, why is it so hard to teach the thing we want to a model than to a human? For a human, we don’t necessarily need a verifiable reward to be able to… You’re probably mentoring a bunch of researchers right now, and you’re talking with them, you’re showing them your code, and you’re showing them how you think. From that, they’re picking up your way of thinking and how they should do research. You don’t have to set a verifiable reward for them that’s like, “Okay, this is the next part of the curriculum, and now this is the next part of your curriculum. Oh, this training was unstable.” There’s not this schleppy, bespoke process. Perhaps these two issues are actually related in some way, but I’d be curious to explore this second thing, which is more like continual learning, and this first thing, which feels just like sample efficiency.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.