Jeff Dean: evaluation

Transcript context

…Yeah. Are there any benchmarks or like test sets they use internally? Because it’s almost like the same benchmarks get reported every time. And it’s like, all right, it’s like 99 instead of 97. Like, how do you have to keep pushing the team internally to it? Or like, this is what we’re building towards. Yeah. I mean, I think. Benchmarks, particularly external ones that are publicly available. Have their utility, but they often kind of have a lifespan of utility where they’re introduced and maybe they’re quite hard for current models. You know, I, I like to think of the best kinds of benchmarks are ones where the initial scores are like 10 to 20 or 30%, maybe, but not higher. And then you can sort of work on improving that capability for, uh, whatever it is, the benchmark is trying to assess and get it up to like 80, 90%, whatever. I, I think once it hits kind of 95% or something, you get very diminishing returns from really focusing on that benchmark, cuz it’s sort of, it’s either the case that you’ve now achieved that capability, or there’s also the issue of leakage in public data or very related kind of data being, being in your training data. Um, so we have a bunch of held out internal benchmarks that we really look at where we know that wasn’t represented in the training data at all. There are capabilities that we want the model to have. Um, yeah. Yeah. Um, that it doesn’t have now, and then we can work on, you know, assessing, you know, how do we make the model better at these kinds of things? Is it, we need different kind of data to train on that’s more specialized for this particular kind of task. Do we need, um, you know, a bunch of, uh, you know, architectural improvements or some sort of, uh, model capability improvements, you know, what would help make that better? Is there, is there such an example that you, uh, a benchmark inspired in architectural improvement? Like, uh, I’m just kind of. Jumping on that because you just.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.