Public evidence record

Olivia Watkins

Published podcast speaker

Claims: 3
Episodes: 1
Shows: 1
Named items: 0

Claim ledger

What Olivia said.

3 transcript-backed records

01 / evaluation

23 Feb 2026

I think the most common problem are like overly narrow tests where there’s some particular implementation detail that the tests we’re looking for, but wasn’t specified in the problem description.

“I think the most common problem are like overly narrow tests where there’s some particular implementation detail that the tests we’re looking for, but wasn’t specified in the problem description.”

Speaker: Olivia Watkins
Episode: ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
Publisher: Latent Space

02 / prediction

23 Feb 2026

I think, I mean like 90% of the problems are things that were estimated to take like an expert software engineer like less than an hour.

“I think, I mean like 90% of the problems are things that were estimated to take like an expert software engineer like less than an hour.”

Speaker: Olivia Watkins
Episode: ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
Publisher: Latent Space

03 / disagreement

23 Feb 2026

Progress has kind of stalled. And basically we realized that this is because the eval is effectively saturated and also highly contaminated.

“Progress has kind of stalled. And basically we realized that this is because the eval is effectively saturated and also highly contaminated.”

Speaker: Olivia Watkins
Episode: ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
Publisher: Latent Space

Search evidence