Jeff Dean: evaluation

Transcript context

…Um, I do think like there, there is that, that weird cliff where like, it feels like we’ve done the easy stuff and then now it’s, but it always feels like that every year. It’s like, oh, like we know, we know, and the next part is super hard and nobody’s figured it out. And, uh, exactly with this RLVR thing where like everyone’s talking about, well, okay, how do we. the next stage of the non-verifiable stuff. And everyone’s like, I don’t know, you know, Ellen judge. I mean, I feel like the nice thing about this field is there’s lots and lots of smart people thinking about creative solutions to some of the problems that we all see. Uh, because I think everyone sort of sees that the models, you know, are great at some things and they fall down around the edges of those things and, and are not as capable as we’d like in those areas. And then coming up with good techniques and trying those. And seeing which ones actually make a difference is sort of what the whole research aspect of this field is, is pushing forward. And I think that’s why it’s super interesting. You know, if you think about two years ago, we were struggling with GSM, eight K problems, right? Like, you know, Fred has two rabbits. He gets three more rabbits. How many rabbits does he have? That’s a pretty far cry from the kinds of mathematics that the models can, and now you’re doing IMO and Erdos problems in pure language. Yeah. Yeah. Pure language. So that is a really, really amazing jump in capabilities in, you know, in a year and a half or something. And I think, um, for other areas, it’d be great if we could make that kind of leap. Uh, and you know, we don’t exactly see how to do it for some, some areas, but we do see it for some other areas and we’re going to work hard on making that better. Yeah. Yeah.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.