David Rein: evaluation

Transcript context

…chose this the the measurement being a human who has the background expertise but is new to, like, this specific job or this specific task. It's like that's roughly the sort of level of knowledge we expect models to have so that, you know, they sort of, like, know you know, we basically don't expect them to be bottlenecked on expertise that is available on the public Internet or, you know, things that people could learn in university sort of thing. So, like, you know, they're they're coming in with probably the level of at least the level of knowledge of someone who's sort of an expert in the right discipline, but they won't know that company specific software or the, you know, sort of this exact problem before. So that's, like, hopefully, it's sort of roughly the right analogy. You know, I do think that, like, to the extent people, interpret the kind of takeaway numbers as like, oh, yeah, Opus 4.6 can do anything that I do in my job that takes me 12 hours or whatever. To the extent that people have that takeaway, I think that is almost definitely an overestimate, for example, because of this issue where, yeah, when you're doing a 12 hour task in your job, you could not easily delegate that to a human contractor. It would take them maybe like weeks or something to do a task like that. And just quickly, where did you find the people? Do I understand that some of them were contractors and some of them were employees? And…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.