Sholto Douglas: belief

Transcript context

…You teach them to solve a particular coding problem, but the thing you've taught them is just “write all the code you can to make this one thing work.” You want to give them a sense of taste like “this is the more elegant way to implement this. This is a better way to write the code, even if it's the same function”. Especially in writing where there's no end test, then it's just all taste. How do you reduce the slop there? I think in a lot of these cases you have to hope for some amount of generator verifier gap. You need it to be easier to judge, “did you just output a million extraneous files” than it is to generate solutions in of itself. That needs to be a very easy to verify thing. So slop is hard. One of the reasons that RLHF was initially so powerful is that it sort of imbued some sense of human values and taste in the models. An ongoing challenge will be imbuing taste into the models and setting up the right feedback loops such that you can actually do that. Here's a question I'm really curious about. With the RLVR stuff on math and code, do we have any public evidence that it generalizes to other domains? Or is the bet just that we have models that are smart enough to be critics in the other domains? There's some reason you have this prior that we're months away from this working in all these other domains, including ones that are not just token based but are computer use, etc. Why?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.