Tim Scarfe: evaluation

Transcript context

…I agree. There was 3 of us looking at it and we we just it's and 1 of those things that depending on your perspective, you might get it straight away or or you might not. So there's that criticism. And people have said that ARC v 3 is even harder. Yeah. Yeah. You know, but I I think that's rather missing the point. I I think he's saying that with with with a lot of these competitive coding problems, The the dataset is contaminated. These are problems that have been solved before in in part or in whole, which means when you look at the epistemic tree, many of the building blocks for solving them are very high up in the tree. He's he's looking at these these problems that there is very little dataset contamination. And they need to be solved from very abstract building blocks. So you're starting much lower down the tree and you're synthesizing a model by composing together very abstract building blocks, is the essence of intelligence. Yeah. And and I think for that reason, ARC is is really kind of pushing us to build adaptive systems which we could say are intelligent. Yeah. I agree. I I mean, like, in many ways, I'm I'm really looking forward to the next years and seeing how far we can push this and then also how much generalization we can get afterwards. Because I I I believe, like, when you look at sort of the more recent models, they're getting much better at the transform style code evolution or outputting for ARC than they are on the instruction based level. And I think this might already be like a small sign of some amount of overtraining on ARC AGI 1 at least. Right? I do believe there are some aspects of work which will be automated before it comes to sort of fully science automation and the type of work I'm doing. But I could imagine that certain parts of the dimensions that I deal with every day are for sure going to be hit by AI. And then the question is, are there gonna be new dimensions opened up that we as humans will fill in? Right? And I think what I said before about, like, shepherding and so on, I really hope that that's the way forward, right, in the sense that humans are the ones steering the ship while just being massively amplified in their productivity.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.