Evidence receipt / belief
Published · transcript-backedGary Marcus: belief
24 Jun 2025 Machine Learning Street Talk Three Red Lines We're About to Cross Toward AGI (Daniel Kokotajlo, Gary Marcus, Dan Hendrycks)
“I think if you rule out data contamination, the performance on problems that are new is not good.”
Source trail
Everything needed to verify it.
- Speaker
- Gary Marcus
- Attribution
- Verified speaker
- Claim type
- belief
- Recorded
- 24 Jun 2025
- Publisher
- Machine Learning Street Talk
Transcript context
…Not much. Tell me about it. So I don't know what the human data on it, but I know that the machine data on it was 0%. And what was interesting about the LiveCode Pro benchmark LiveCode Bench Pro, excuse me, was that they were all brand new problems. And so they ruled out data contamination. And I've seen 2 careful studies ruling out data contamination. The other was with the US AMO, US Math Olympiad problems 6 hours after testing. Performance was terrible on them. I think if you rule out data contamination, the performance on problems that are new is not good. And for programming, and especially programming AGI, that's super relevant. You know, you're using these things to code up a website, that's not new. There's so many examples. But if you're using it to do something that's new, you get out of distribution, there's still a lot of problems. What about frontier math? Wasn't that also…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.