Jeff Dean: prediction

Transcript context

…immediately everyone jumped to like completely green charts of like, everyone had, I was like, how did everyone crack this at the same time? Right. Yeah. Yeah. I mean, I think, um, and once you’re set, I mean, as you say that needed single needle and a half. Hey, stack benchmark is really saturated for at least context links up to 1, 2 and K or something. Don’t actually have, you know, much larger than 1, 2 and 8 K these days or two or something. We’re trying to push the frontier of 1 million or 2 million context, which is good because I think there are a lot of use cases where. Yeah. You know, putting a thousand pages of text or putting, you know, multiple hour long videos and the context and then actually being able to make use of that as useful. Try to, to explore the über graduation are fairly large. But the single needle in a haystack benchmark is sort of saturated. So you really want more complicated, sort of multi-needle or more realistic, take all this content and produce this kind of answer from a long context that sort of better assesses what it is people really want to do with long context. Which is not just, you know, can you tell me the product number for this particular thing? Yeah, it’s retrieval. It’s retrieval within machine learning. It’s interesting because I think the more meta level I’m trying to operate at here is you have a benchmark. You’re like, okay, I see the architectural thing I need to do in order to go fix that. But should you do it? Because sometimes that’s an inductive bias, basically. It’s what Jason Wei, who used to work at Google, would say. Exactly the kind of thing. Yeah, you’re going to win. Short term. Longer term, I don’t know if that’s going to scale. You might have to undo that.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.