Josh Albrecht: belief

Transcript context

…I want to touch on maybe a couple of hot topics in evals, maybe tangentially related, but we're on the evals train right now. So I'm just going to get on that. So ArcAGI, Francois Chollet's hot new thing, it's sort of my take on it is basically it's trying to measure reasoning through an abstract IQ test. Effectively, I noticed that you don't use it. There's a lot of community debate, pro and con about it. What are your thoughts on just more abstract reasoning and maybe ArcAGI specifically? I think we purposely stayed away from the very, like there's BigBench, for example, that has a lot of, I think, to me, feels sort of similar types of tasks that are like very unrealistic. Like, oh, you know, we have books of different colors and then you're going to shuffle them and like which book is furthest to the left or something like, OK, cool, I guess it's neat. It's neat, I think, for us to explore in terms of like an agent reasoning in a larger loop. And we do care about these types of evaluations there. The types of evaluations we're talking about in the blog post here are for getting at, like, does this model in a base model sense, is this working at all? There's no chain of thought in these evaluations. These are just like, go straight to the answer. Does this make sense? Like, is this a thing that…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.