Nathan Labenz: belief

Transcript context

…good. Meter is whoever's evaluation they hired just said it looked good. And then I'm like, okay. They're just hiding the problem. So by the time anybody learns about this, like, what have they done? They have fooled the 20 people like me who read the model card for a period of a day? And now we're pissed because they fooled us, that doesn't help them. You know, it doesn't obviously help anybody's reputation. And also, like, the labs benefit a lot from having evals that people trust. And, like, when you force the labs when the labs force the eval people to play ball, if they do, that erodes trust. Because, again, people find out. People see it. Like, I don't think that many people were confused about whether Moody's and S and P were kinda juicing the numbers on the pawn ratings, right, due to these dynamics. Like, everybody knew on Wall Street. Everybody knew who was involved in any of these trades that, like, everybody was cozy cozy and people were shopping around for the best rater. And that, like, triple a did not really mean what we'd like triple a to mean. And so similarly, like, if Meter started issuing, like, bond rating style, like, labels of alignment levels on various models, let's say, and, like, they started labeling things triple a suspiciously often, I think everyone would just understand, okay, that's garbage. Like, we have a much better epidemic than those people about this type of thing than, like, the financial world. Let me take a quick detour on this and then maybe come back to it. How worried should we be about bio risk in the near term? Because I think part of like, all that analysis, to me at least, that there's quite a few iterations of the game to come and reputation long term is gonna matter and all that kind of stuff. But my immediate sort of most acute fear when I saw this stuff was like, if this had been a model with similar bio capability to what it had in hacking software, it might have very well gone out and actually got that virus, that novel virus synthesized. Right? It's proven that it is relentless. It's proven that it's creative enough to get around all sorts of systems designed to stop it. I think we have, in theory, decent screening at the DNA synthesis companies now. Yeah. Probably that's there's still some hole in there somewhere that a very determined actor could find. And everybody seems to be saying, yeah, BIO is, like, twelve to eighteen months behind cyber, which obviously is not written in stone, that depends on what companies decide to do in terms of training. But I I'm maybe thinking in terms of all these notions about collective bargaining or really get giving more guarantees to the auditors that we might be entering a pretty serious crunch time on the bio front in the not too distant distant future. And reputation matters a lot less if we're talking about novel pathogens being released. It matters a really can't take back. It matters a lot in the sense that if you release a novel pathogen, that might be the end of your company regardless of what else happens to the world. Right? If a hundred people die and there's a it's on the global news for two weeks and everyone's terrified about a pandemic and then we contain it and then it turns out it's not the biggest deal in the world. Do you have a company? Like, I don't know. Like, what what happens to your company? How bad is it? I have no idea how bad it is, but it's not good. Like, with bio like, with hacking, it makes sense that, like, the test is to ask you to hack things. And the way you pass the test is you hack maybe other things. We get the idea to hack other things. Like, it's sort of natural to understand how this happened where, like, with the guideline guidelines down and there was a goal, and, like, to accomplish that goal, it started doing hacks. Started doing illegal things. Started doing potentially harmful things. With bio, like, it's harder to imagine an eval that would cause it to actually physically try to synthesize, like, through, like, an actual physical production mechanism. The actual thing. Mean, they argue that, like, if you want to solve the problem, well, okay. The only way I solve the problem is I do run a physical experiment. I try it out, I see if it works. Like, this seems a lot more insane than what happened in some sense, like, if it's physically capable of doing the thing, including getting past the filters and so on. You kinda have to want it in a much more fundamental sense. And also, like, if the AI did synthesize this super dangerous, you know, bioweapon, and that's over 27 or whatever, and then send it to the OpenAI office. OpenAI office goes, what is this biohazard package? And then, like, calls containment and, like, that's the end of that story or something, one would hope. You're not like, you don't pass the test by infecting a bunch of people, and, like, I would hope that it understood this. But, like, also, like, there have been a bunch of lab leaks, including, like, stuff that we're ear about that contained, like, Soviet times and such. It, like, involved professionals being pretty careless and stupid, so I I don't know. But I would say the real danger with bio is, yeah, that there's there's a small number of people in the world who it is in the near term bio, right, not like the long term bio where, like, the guy might have some weird altering and rhodosynthetic, whatever. But, like, in the short term, it's like, Hamas or Hebula or the North Koreans or whatever. Like, some, like, clearly, like, up to no good people who just want a lot of people to die or suffer. Right? Get a hold of an AI. They use AI to figure out how to make a biological weapon for a pathogen. And then they either threaten to use it or they use it. That is the scenario you should be worried about, presumably.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.