High Signal Podcasts Evidence ledger
Method
Browse
← All source episodes

Latent Space / episode intelligence

⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data

23 Feb 2026 11 published claims 3 attributable people

Speakers in the public record

Claim mix

evaluation 6belief 1commitment 1preference 1prediction 1disagreement 1

Evidence policy

Every row below preserves an exact excerpt. Identified speakers are linked; unresolved voices are labeled and excluded from people counts.

Claim ledger

The useful parts, with receipts.

11 published records

01 / belief

I think the let’s, let’s sort of reset on like, what was the original work that you guys did for Verified, which I think was pretty substantial.

“I think the let’s, let’s sort of reset on like, what was the original work that you guys did for Verified, which I think was pretty substantial.”
Speaker
Shawn Wang
Publisher
Latent Space

02 / commitment

I will note that you guys had a trend in verifying benchmarks. ‘cause I just recently saw, I think Quinn had a HLE verified for humanities.

“I will note that you guys had a trend in verifying benchmarks. ‘cause I just recently saw, I think Quinn had a HLE verified for humanities.”
Speaker
Shawn Wang
Publisher
Latent Space

03 / evaluation

I, I think you guys have, you know, really made a lot of progress and I think taking a lot of industry leadership for C Bench verified and, and now moving on to C Orange Pro.

“I, I think you guys have, you know, really made a lot of progress and I think taking a lot of industry leadership for C Bench verified and, and now moving on to C Orange Pro.”
Speaker
Shawn Wang
Publisher
Latent Space

05 / evaluation

Where, where that’s a significant, it like that means that if you pass a test, actually, like you probably did like a really good job, but just because you didn’t pass a VE test [00:08:00] doesn’t mean that your implementation wasn’t like a good one.

“Where, where that’s a significant, it like that means that if you pass a test, actually, like you probably did like a really good job, but just because you didn’t pass a VE test [00:08:00] doesn’t mean that your implementation wasn’t like a good one.”
Speaker
Mia Glaese
Publisher
Latent Space

06 / evaluation

I mean, we had to do it, we had to do it actually, because it’s quite a hard task to like look at something like a, a problem and, and the patch and then like, it’s not just the problem and the patch, right?

“I mean, we had to do it, we had to do it actually, because it’s quite a hard task to like look at something like a, a problem and, and the patch and then like, it’s not just the problem and the patch, right?”
Speaker
Mia Glaese
Publisher
Latent Space

07 / evaluation

I think the most common problem are like overly narrow tests where there’s some particular implementation detail that the tests we’re looking for, but wasn’t specified in the problem description.

“I think the most common problem are like overly narrow tests where there’s some particular implementation detail that the tests we’re looking for, but wasn’t specified in the problem description.”
Speaker
Olivia Watkins
Publisher
Latent Space

09 / evaluation

I think like complexity, however we can sort of like quantify it, is really important to understand like where our motive are, are are getting to.

“I think like complexity, however we can sort of like quantify it, is really important to understand like where our motive are, are are getting to.”
Speaker
Mia Glaese
Publisher
Latent Space

10 / evaluation

Like when they start to become like popular and like viable, it’s because they measure something like important and modes maybe do like 20% correct on them, sometimes even less.

“Like when they start to become like popular and like viable, it’s because they measure something like important and modes maybe do like 20% correct on them, sometimes even less.”
Speaker
Mia Glaese
Publisher
Latent Space

11 / disagreement

Progress has kind of stalled. And basically we realized that this is because the eval is effectively saturated and also highly contaminated.

“Progress has kind of stalled. And basically we realized that this is because the eval is effectively saturated and also highly contaminated.”
Speaker
Olivia Watkins
Publisher
Latent Space
Search evidence