High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Erik Schluntz

Published podcast speaker

Claims
28
Episodes
1
Shows
1
Named items
0

Claim ledger

What Erik said.

3 transcript-backed records

01 / evaluation

The question is, are those fairly inaccessible or are they just impossible because of the descriptions? But I think certainly some of the tasks, especially the ones that the human graders reviewed as like taking longer than four hours are extremely difficult.

“The question is, are those fairly inaccessible or are they just impossible because of the descriptions? But I think certainly some of the tasks, especially the ones that the human graders reviewed as like taking longer than four hours are extremely difficult.”
Speaker
Erik Schluntz
Publisher
Latent Space

02 / evaluation

The other thing I was going to say is that SWE-Bench is certainly hard to implement and expensive to run because each task, you have to parse, you know, a lot of the repo to understand where to put your code.

“The other thing I was going to say is that SWE-Bench is certainly hard to implement and expensive to run because each task, you have to parse, you know, a lot of the repo to understand where to put your code.”
Speaker
Erik Schluntz
Publisher
Latent Space
Search evidence