Beth Barnes: observation

Transcript context

…you know, distributional shift between the benchmark and the real world. And the the only reason I asked that question is, I mean, I'm sure you folks, you you probably struggle to hire people. It's really difficult to hire people. So, like, if if you're getting people to solve very challenging problems, it's not like you can just go out there and just grab competent people. It's it's very, very difficult. At some point, for RE bench baselines in particular, we got more, you know, a large number of benchmarks per question, and we were looking at people's, like, qualifications and years of experience and things. We actually ended up with a negative correlation between years of experience and, and and performance because, like, the the sort of people who are in network, like, our friends who are kind of doing really well, and the people who are more qualified were, like, actually not doing that great. So, yeah. You know, it's, it's tricky. Well well, yeah. Exactly. I mean, I don't wanna spend too long on this, but I I have similar intuitions. I I think that knowledge is perspectival. It's quite path dependent. So you're gonna find people in group that are just culturally thinking about things in the same way because when we do have these abstract notions of skill, like, oh, they have a PhD, they have this many years of experience, it's actually not a very good reflection. So there is a bit of a thing here about using abstract…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.