Gary Marcus: evaluation

Transcript context

…It won't be like a sharp cutoff, but it'll be a couple of things. So so partly, it'll be, you know, power supplies. Partly, it'll just be, compute production, like much of the world's even after building new fabs and even after converting much of the world's chip production into AI chips, they'll have to produce 10 times more fabs in order to scale up by 10 times, right? Whereas previously, they could just take chips designed for gaming and repurpose them for AI. So in a bunch of little ways that are going to add up, there's going to be all these frictions that's going to start to bite that will make it harder for them to continue the crazy exponential rate of scale up to the I think we're already seeing that with data. I don't know the actual numbers. But, let's say that GPT 2 used maybe 10% of the internet or something like that or 5% or something like that. Maybe you guys know the actual numbers. And GPT 3, you know, used a significantly larger fraction. GPT 4 used like most of the internet including transcriptions of of videos and stuff like that. And so you can't just keep a 100 x ing that because there just isn't enough data. There's new data generated every day, you know, so you can always eke out a little more and people are turning to augmented data as well they should. But that's not a kind of universal solvent. It works better for things like math where you can verify that the the augmented data are better. And so I think we're already running against that, kind of like bottleneck on 1 of the, let's say, raw resources that go in at least into the current approaches. And another thing I would add is that, you can also just think about money, which is, which you can use to buy many of these things. And you can say, well, they've been scaling up the amount of money that they're spending on AI research and on training runs in particular, over the last decades. But they it'll be hard for them to continue scaling at the same pace. Know? They they're probably already doing something like billion dollar training runs. But, if it goes at the same pace the biggest training run-in 2020 was like, what, dollars 3,000,000, something like that, dollars 5,000,000? So they've gone up by 2 and onetwo orders of magnitude in 5 years. If it's another 2 and a half orders of magnitude, we're doing a $500,000,000,000 training run-in 02/1930,…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.