Ryan Kidd: recommendation

Transcript context

…One of the things that I took note of in the blog post from 18 months or so ago was you had made a comment that funders basically don't want to, or they're much more inclined to support the growth of organizations that they sort of see as legible, that have like research directions that they sort of feel are somewhat established or that they can wrap their heads around. And they're much more reluctant to fund like totally new conceptual directions. And that seems like it exists in contrast with like the AE Studio survey, where they basically found that the field as a whole seems to think that like, we don't have all the ideas that we need and, you know, that like more kind of far out ideas should be tried, which of course led to their neglected approaches approach. What do you make of that? Is there stuff that we can do or is it, you know, is there is it a different organization's job to figure out how to fill that gap. Because I do feel like I want some more, and I love some of the AE Studio stuff, including self-other overlap. I always come back to that as an example of something that's just quite off the map of what most people are doing. When I think of AI control and what Buck and the Redwood Research team are doing, I find that stuff fascinating. And one of the things that kind of impresses me most is that they are willing to work on something that in some ways is so depressing. They're like, we're going to try to figure out how to work with AIs, even assuming they're out to get us. And I'm like, yikes. I don't know that I would be able to sustain the positive attitude enough to do that if I was working from that premise. I do feel like there's a relative dearth of things that are more inspiring. Here, I think maybe of AI Studio, but also Softmax. Obviously, people have a lot of different opinions on, are these things ever going to work or not? I wonder what your take is on just kind of the overall mix. It seems like a lot of things are kind of more toward patch the holes, keep the AI down, tempt it, you know, see if it'll take the temptation, and then patch it, you know, if it takes the temptation. And there's not nearly as much that is sort of a, a more kind of colorful, positive vision for the future. And I wish there was, but maybe that's just not happening because The ideas are just too hard to come by. Maybe it's not happening because the funders aren't bold enough. What's your take on how we can get, if we should be trying to get more of that stuff? And if you think we should, how might we go about it? I have many takes here. So obviously I advocate a portfolio. And Matt's has historically sponsored a bunch of projects. Self-other overlap, that project came out of maths alum, I'll just say Mark Carlineau, I might have messed up his name, was like the originator of that project at AA Studios. And I believe Cameron Berg is running, another maths 1.0 alum with me, is running some of their more neuroscience-inspired approaches as well. So ASuers is great. I love what they're doing. I think that the survey they did have left wrong is just like probably not representative of the AI safety research field as on the whole. But then it might be. Even so, I think we obviously need more ideas because more ideas are good, right? More bets are good. More shots on goal are good. Now, I would not advocate a person who is like a very strong iterator to drop that and try and become think of some new paradigm that is I think that would be strictly counterproductive. on the margin because I think we do have some very strong central research bets that we need more people pursuing because they will yield demonstrable results. But if everyone did that, this would be bad because you need to have your portfolio. Maybe these approaches fail. Maybe they need other pieces to work. Many AI safety research agendas are kind of contingent on other things going right or other people working on other stuff. It's like any kind of research field. You need to have everyone advancing the frontier. So I think High safety has historically gone really kind of argmaxy on different agendas, which is bad. Portfolio approach is much better. Don't rule things out as possible directions. Just shift and reallocate resources to them. To their credit, Coefficient Giving have done an amazing job particularly recently, at supporting a bunch of different novel research bets. And they've also funded PIBs, or Principles of Intelligence, a program that is trying very hard to pursue sort of moonshotty interpretability and agency understanding projects. So they're great. Check them out. I think that more ideas would be good. I think that the kind of person who should be pursuing that typically is going to look something like someone who is already a domain expert in some other area. You are occasionally going to have your Buckschlegeresses, your Evan Hubingers, right, who come along with no PhD, but spent years at MIRI, you know, incubating in that kind of, that deep AI safety, that rich safety experience, and then come out with amazing stuff like, risk and learner optimization and AI control and all that stuff. But short of, you know, having access to that type of community and that type of research experience, I think most of the prominent connectors, like your Alex Turners and so on, have come, have spent a lot of time in research science PhDs, also on Vestrong, of course, and like, you know, incubating in that environment as well. ent connectors, like your Alex Turners and so on, have come, have spent a lot of time in research science PhDs, also on Vestrong, of course, and like, you know, incubating in that environment as well. So I think MATS is a great way for that kind of person to develop and to spawn more research ideas. In fact, I've seen To shout out Alex Turner. He has come up with some amazing research ideas over his time at MATS. And I think we've been very fortunate to support him. Things like gradient routing, and also Alex Cloud, another MATS mentor, and just plenty of other things, like activation engineering and steering. He was one of the people involved in that. So I think that senior experienced researchers are going to be probably, like most things, the main drivers of new ideas. and grant funding that lets them pursue whatever their research taste dictates is great. And programs like Matt's that let them stalk their research agendas are also great. I also think bounty programs could work as well, but I would hazard against people putting all their eggs in the basket of we need to have a bunch of new ideas because the central idea is not working. I don't think that's true. I think the central ideas are still our actual best bets.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.