Steve Newman: evaluation

Transcript context

…How close is Mythos to Jeff Dean, though? Do you have a-- Sorry? How close is Mythos to Jeff Dean? I only know, you know, what they said in the model card and whatever. I think still not very close. Actually, let me come back to that. I want to talk about threshold effects for a second, then I'll come back to that. Sure. Yeah, I don't know what, like, specific next thresholds I'm looking for. I can't guess what it's going to be, but I was thinking about this a little bit. We all talk about AI capabilities and like, oh, 4.7 just dropped, what new thing is it gonna be able to do and whatever. But we don't, all of us in our day-to-day, as we're engaging with these tools and as we're living through the impact of these tools on the world and the environment we're in, whatever, we're not really engaging with model capabilities, we're engaging with a whole complicated ecosystem of what could the model do if you prompted it well, and how well are people prompting it, and who's using it and who isn't, and what second and third and fourth order implications does that have? And I think this is where the threshold effects come from. You know, like the ChatGPT launch was a moment. Multbot was a moment. not because that was the day that the, you know, the scaling curve crossed some threshold, but because that was the contingent moment when capabilities had gotten far enough and there was a little bit of an overhang and then someone happened to do something and they happened to do it in a way that caught people's attention. And yeah, I think it's, maybe it's a little bit like there have been coronaviruses circulating in bats that like, have an R of 0.9 in human beings, like every now and then one will cross to a human being and maybe infect two or three more people and peter out. And then one day there happened to be one that had an R of 1.1 and infected a couple more people and a couple more, and then it was evolving and getting better at spreading in humans. And so that was a threshold effect where suddenly COVID exploded through the human population. but only because of the dynamic effect. It wasn't about what that virus did in one person, it was about the way it went from person to person to person. It was the forward evolving system of people in virus that had tipped over into a new domain. And that's just, and like on a more complicated level, That's what's happening with AI. You know, like a lot of people, I'm part of a big cohort of people who started Claude coding in December because 4.5 was out and we had, and it was the December break. And yeah, like a lot of this, like it wasn't just because of like a specific new capability. It was, you know, I was reading other people saying that 4.5 is worth trying. So. Going back to Mythos and Jeff Dean, and maybe generalizing the question a little bit, obviously we haven't, either we haven't or we're sworn to secrecy in our access to Mythos. I haven't. You may be sworn to secrecy.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.