Public evidence record
Published podcast speaker
Claim ledger
6 transcript-backed records
01 / recommendation
“yeah, I think you have to measure the model's or the system's ability to generalize to kind of actually novel situations.”
02 / belief
“At least me personally, I think we don't have these results aren't like kind of collected and we don't have kind of as systematic results as we do compared to like time horizon partially because, you know, scoring is is is is qualitative now, for for these tasks.”
03 / belief
“I I and and, yeah, I think, you know, are a lot of different, kind of perspectives or or kind of prior beliefs people people can have that, you know, I think there's a wide range of kind of reasonable, judgments about where we're going to be.”
04 / evaluation
“To the extent that people have that takeaway, I think that is almost definitely an overestimate, for example, because of this issue where, yeah, when you're doing a 12 hour task in your job, you could not easily delegate that to a human contractor.”
05 / evaluation
“place to start for us in terms of the motivation for the time horizon work is to have a unified axis that we can measure AI progress on over a very long period of time. So when we started doing the work, we had this very strong belief that GPT-two is fundamentally, in some really important sense, much worse as an AI than, I guess, yeah, at the time it was maybe Summit 3.”
06 / evaluation
“You know, there there is kind there there are kind of like standard results in in in economics where, you know, if you make you know, as as you if you if you automate like a small fraction of some some labor market, then it can be the case that actually, yeah, it becomes more profitable to work in that market because you're more productive, which I think is kind of how I understand what's happening now.”