Tim Scarfe: disagreement

Transcript context

…Right? All of those endless permutations to to RNNs were suddenly seemingly a waste of time. We're kind of in a situation right now where a lot of the papers are just taking the same architecture and making these endless amount of different tweaks of, like, you know, where to put the normalization layer and slightly different ways of training them, and and we might be wasting the time in exactly the same way. Right? Like, I personally don't think we're done. Right? I don't think that this is the final architecture, and we just need to keep scaling up. There's some breakthrough that will occur at some point, and then it will once again become obvious that we're kind of wasting a lot of time right now. Yeah. So we are a victim of our own success. And this basin of attraction, there are so many basins of attraction. Sarah Hooker spoke about the hardware lottery. And this is a kind of architecture lottery. And it actually made me think of the agricultural revolution, which is that this kind of phase change happened. And all of the folks that had these skills that were so necessary, these diverse skills for living and surviving, they died out. And that's actually quite paradoxical because we need those skills to take the next step. And so we're now in this regime. We've got the term foundation model. And the implication is that you can do anything with a foundation model. In the corporate world, we used to have data scientists. Know, there were ML engineers doing these architectural tweaks even in, you know, mid sized enterprise. And now we just have AI engineers who are just doing prompt engineering and so on. So you're saying that the fundamental skills that we need to be diverse, to think of new solutions and new architectures, they're dying out. I think I'm gonna disagree with that. I think the problem is we have plenty of very talented, very creative researchers out there, but they're not using their talents. Right? For example, you know, if you're in academia, there's pressure to publish. Right? And if there's pressure to publish, you think to yourself, okay, well, I have this really cool idea, but it might not work, it might be too weird, right, it might be difficult to get it accepted because I have to sort of, like, sell the idea more. Or I can just try this new position embedding. Right? The problem is that the current environment, both in academia and in companies, are not actually giving people the freedom that they need to do the research that they probably want to do.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.