Evidence receipt / evaluation
Published · transcript-backedErik Schluntz: evaluation
28 Nov 2024 Latent Space The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
“The question is, are those fairly inaccessible or are they just impossible because of the descriptions? But I think certainly some of the tasks, especially the ones that the human graders reviewed as like taking longer than four hours are extremely difficult.”
Source trail
Everything needed to verify it.
- Speaker
- Erik Schluntz
- Attribution
- Verified speaker
- Claim type
- evaluation
- Recorded
- 28 Nov 2024
- Publisher
- Latent Space
Transcript context
…Yeah, like specifically, is there a category of problems that are still unreachable by any LLM agent? Yeah, yeah. And I think there definitely are. The question is, are those fairly inaccessible or are they just impossible because of the descriptions? But I think certainly some of the tasks, especially the ones that the human graders reviewed as like taking longer than four hours are extremely difficult. I think we got a few of them right, but not very many at all in the benchmark. And did those take less than four hours?…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.