Zvi Mowshowitz: belief

Transcript context

…Do you have a best case scenario for what safe superintelligence might be up to? Could you imagine anything that feels both somewhat realistic and, wow, that would be amazing if they could come forward with something like what you have in mind? I think I think agent vanishingly is probably the best case scenario. Maybe they've been doing a bunch of theoretical work on a different architectural approach that is more like reliability bound and that has more like, it is much less of a weird black box of soup and that allows you to steer it and guide it better. Certainly seems like there's no reason it couldn't exist. That would be my top hope. Obviously, it could also be a fear if it's, like, even worse. But again, like, they've they don't talk, so we don't know. Yeah. I just I have no idea. I know that Ilya's statements in interviews on the problem space did not seem to be that well informed about, like, how the problems work that I am worried about. You know, I'm not necessarily sure he would be able to differentiate approaches that would be better at solving those problems than approaches that wouldn't be. But, you know, I'd be very curious. I can safely say that nobody's telling me anything. The invitation is open to the podcast, Ilya, if you're listening. For what it's worth, a vision that I have that I think could be pretty cool. Thinking Machines is doing kind of a version of this with their foundation model that's really designed to be fine tuned. But I was thinking a paradigm where, like, a stem cell where you create something that is a proto intelligence that can be adapted to a wide variety of circumstances and can get really good at doing its job in a particular niche. But in the process of getting good at that sort of trades away or prunes away the the super broad capabilities that the current models have so that you have something that's, like, small, fits its role well, does a good job, but can only do that thing in the way that once you go from a stem cell to a specialized cell, like, it has this particular role that it does it can fulfill effectively, but it's not gonna go off and do other things. Obviously, there's some problems in that analogy. But I think that could be a a I would love to see somebody pursue that kind of rather than scaling up always bigger, better, more. Can we create something that settles and fits into its place? This is very much like a Drexler reframing superintelligence vision from years ago as well. Two techniques that have recently come out that I wanna get your reaction to. One is GRAM, gradient routing. I forgot what the a m stands for, but the the promise of it is to localize certain kinds of knowledge to particular experts within an MOE architecture so that you can hopefully have your cake and eat it too in terms of distributing, maybe even open source or open weights at least, the model minus the experts of concern while potentially having also a structured access program for your trusted biologists or what have you. Right. I'm excited about it, but I wouldn't be doing my job if I didn't give you the chance to pour some cold water on it.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.