Dwarkesh Patel: uncertainty

Transcript context

…I wouldn’t say there was one big surprise. It was very interesting trying to train things at that size and learning about all sorts of things from an organizational standpoint, like how to babysit such a system and to track it. There’s also things like getting a better understanding of the metrics you’re optimizing versus the final capabilities that you want. I would say that’s still not a perfectly understood mapping, but it’s an interesting one that we’re getting better and better at. There’s a perception that maybe other labs are more compute-efficient than DeepMind has been with Gemini. I don’t know what you make of that perception. I don’t think that’s the case. I think that actually Gemini 1 used roughly the same amount of compute, maybe slightly more, than what was rumored for GPT-4. I don’t know exactly what was used but I think it was in the same ballpark. I think we’re very efficient with our compute and we use our compute for many things. One is not just the scaling but, going back to earlier, more innovations and ideas. A new innovation, a new invention, is only useful if it can also scale. So you need quite a lot of compute to do new invention because you’ve got to test many things, at least some reasonable scale, and make sure that they work at that scale. Also, some new ideas may not work at a toy scale but do work at a larger scale. In fact, those are the more valuable ones. So if you think about that exploration process, you need quite a lot of compute to be able to do that. The good news is we’re pretty lucky at Google. I think this year we’re going to have the most compute by far of any sort of research lab. We hope to make very efficient and good use of that in terms of both scaling and the capability of our systems and also new inventions.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.