High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Carl Shulman

Published podcast speaker

Claims
45
Episodes
2
Shows
1
Named items
0

Claim ledger

What Carl said.

45 transcript-backed records

01 / prediction

If there's no premature AI action, we're building the tools and mechanisms and infrastructure for the takeover to be just immediate because effective industry has to be under AI control and robotics.

“If there's no premature AI action, we're building the tools and mechanisms and infrastructure for the takeover to be just immediate because effective industry has to be under AI control and robotics.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

02 / prediction

Although in this scenario all the AIs in the world have been subverted. They are going along with us in such a way as to bring about the situation to consolidate their control because we've already had the failure of cyber security earlier on.

“Although in this scenario all the AIs in the world have been subverted. They are going along with us in such a way as to bring about the situation to consolidate their control because we've already had the failure of cyber security earlier on.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

03 / evaluation

This whole procedure of humans providing compute and supervising the thing and then building new technologies, building robots, constructing things with the AI's assistance, that can all proceed and appear like it's going well, appear like alignment has been nicely solved, appear like all the things are functioning well. And there's some reason to do that because there's only so many giant server farms.

“This whole procedure of humans providing compute and supervising the thing and then building new technologies, building robots, constructing things with the AI's assistance, that can all proceed and appear like it's going well, appear like alignment has been nicely solved, appear like all the things are functioning well. And there's some reason to do that because there's only so many giant server farms.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

04 / belief

You find this extremely skewed distribution and I found that was really a valuable benefit of doing those deep dive investigations into many things in a systematic way because now I can answer a loose agnostic who knows and all the all this nonsense by diving deeply.

“You find this extremely skewed distribution and I found that was really a valuable benefit of doing those deep dive investigations into many things in a systematic way because now I can answer a loose agnostic who knows and all the all this nonsense by diving deeply.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

05 / belief

Yeah, when you had Eliezer on in the earlier episode he talked about nanotechnology of the Drexlerian sort and recently I think because some people are skeptical of non-biotech nanotechnology he's been mentioning the semi-equivalent versions of construct replicating systems that can be controlled by computers but are built out of biotechnology.

“Yeah, when you had Eliezer on in the earlier episode he talked about nanotechnology of the Drexlerian sort and recently I think because some people are skeptical of non-biotech nanotechnology he's been mentioning the semi-equivalent versions of construct replicating systems that can be controlled by computers but are built out of biotechnology.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

06 / uncertainty

One general heuristic is to find ways to hew closer to things that are rich and bodies of established knowledge and less impenetrable–I don't know how you've been navigating that so far but learning from textbooks and the things that were the leading papers and people of past eras I think rather than being too attentive to current news cycles is quite valuable.

“One general heuristic is to find ways to hew closer to things that are rich and bodies of established knowledge and less impenetrable–I don't know how you've been navigating that so far but learning from textbooks and the things that were the leading papers and people of past eras I think rather than being too attentive to current news cycles is quite valuable.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

07 / commitment

Given the AI's situation, how effective do these things look? And we won't, for example, know what are the particular zero day exploits that the AI might use to hack the cloud computing infrastructure it's running on.

“Given the AI's situation, how effective do these things look? And we won't, for example, know what are the particular zero day exploits that the AI might use to hack the cloud computing infrastructure it's running on.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

11 / belief

I've been emphasizing methods that involve less in the way of technological innovation and especially things where there's more doubt about whether they would work because I think that's a gap in the public discourse.

“I've been emphasizing methods that involve less in the way of technological innovation and especially things where there's more doubt about whether they would work because I think that's a gap in the public discourse.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

12 / belief

I wound up mostly focused on AI but there have been other things that have been raised as candidates and people sometimes say, I think falsely, that this is just another doomsday story there must be hundreds and hundreds of those.

“I wound up mostly focused on AI but there have been other things that have been raised as candidates and people sometimes say, I think falsely, that this is just another doomsday story there must be hundreds and hundreds of those.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

13 / evaluation

If we can probe the thoughts and motivations of an AI and discover wow, actually GPT-6 is planning to takeover the world if it ever gets the chance. That would be an incredibly valuable thing for governments to coordinate around because it would remove a lot of the uncertainty, it would be easier to agree that this was important, to have more give on other dimensions and to have mutual trust that the other side actually also cares about this because you can't always know what another person or another government is thinking but you can see the objective situation in which they're deciding.

“If we can probe the thoughts and motivations of an AI and discover wow, actually GPT-6 is planning to takeover the world if it ever gets the chance. That would be an incredibly valuable thing for governments to coordinate around because it would remove a lot of the uncertainty, it would be easier to agree that this was important, to have more give on other dimensions and to have mutual trust that the other side actually also cares about this because you can't always know what another person or another government is thinking but you can see the objective situation in which they're deciding.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

14 / belief

The AIs become smarter than humans, if they're working in enormous numbers more than humans can supervise I think get harder but when I combine the possibility that we get relatively lucky on the motivations of the earlier AI systems, systems strong enough that we can use for some alignment research tasks, and then the possibility of getting that later with AI assistance that we can't trust fully or we have to have hard power constraints and a number of things to prevent them from doing this takeover.

“The AIs become smarter than humans, if they're working in enormous numbers more than humans can supervise I think get harder but when I combine the possibility that we get relatively lucky on the motivations of the earlier AI systems, systems strong enough that we can use for some alignment research tasks, and then the possibility of getting that later with AI assistance that we can't trust fully or we have to have hard power constraints and a number of things to prevent them from doing this takeover.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

15 / observation

Part of the reason is I want to raise these issues, that’s one reason I came on the podcast and then they have the opportunity to actually examine the arguments and evidence and engage with it.

“Part of the reason is I want to raise these issues, that’s one reason I came on the podcast and then they have the opportunity to actually examine the arguments and evidence and engage with it.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

16 / belief

I think the difference is motivation. We sometimes have people appointed as a legal guardian of someone who is incapable of certain kinds of agency or understanding certain kinds of things and the guardian can act independently of them and normally in service of their best interests.

“I think the difference is motivation. We sometimes have people appointed as a legal guardian of someone who is incapable of certain kinds of agency or understanding certain kinds of things and the guardian can act independently of them and normally in service of their best interests.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

17 / commitment

If we consider this world where AI cognitive abilities have been amped up to such an extreme, we should naturally expect that we will have something much much more potent than the AlphaFolds of today and skills that are at the extreme of human biosciences capability as well.

“If we consider this world where AI cognitive abilities have been amped up to such an extreme, we should naturally expect that we will have something much much more potent than the AlphaFolds of today and skills that are at the extreme of human biosciences capability as well.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

18 / belief

I think broadly comparable chunks from us getting things that are putting us in a reasonably good position going into it and then a broadly similar gain from this genuinely terrifying process at the very end, over a few months or hopefully longer, when this kind of automated research is meaningfully helping.

“I think broadly comparable chunks from us getting things that are putting us in a reasonably good position going into it and then a broadly similar gain from this genuinely terrifying process at the very end, over a few months or hopefully longer, when this kind of automated research is meaningfully helping.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

19 / prediction

Moving back to the thing that happened before we built all the infrastructure for the robots to stop taking orders and there's nothing you can do about it because we've already built them.

“Moving back to the thing that happened before we built all the infrastructure for the robots to stop taking orders and there's nothing you can do about it because we've already built them.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

20 / commitment

I think we should as time goes on give AIs moral consideration and a joint Human-AI society that is moral and ethical is a good future to aim at and not one in which you indefinitely have a mistreated class of intelligent beings that is treated as property and is almost the entire population of your civilization.

“I think we should as time goes on give AIs moral consideration and a joint Human-AI society that is moral and ethical is a good future to aim at and not one in which you indefinitely have a mistreated class of intelligent beings that is treated as property and is almost the entire population of your civilization.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

21 / evaluation

Given that this all happens almost immediately people who might otherwise have enjoyed the basic income may object and say no, no, this is no good and they might respond by saying, well something like the subdivision before maybe there's a restriction, there's a distribution of wealth and then when one has a child there's a requirement that one gives them a certain minimum a quantity of resources and one doesn't have the resources to give them that minimum standard of living or standard of wealth yeah one can't do that because of child slash AI welfare laws.

“Given that this all happens almost immediately people who might otherwise have enjoyed the basic income may object and say no, no, this is no good and they might respond by saying, well something like the subdivision before maybe there's a restriction, there's a distribution of wealth and then when one has a child there's a requirement that one gives them a certain minimum a quantity of resources and one doesn't have the resources to give them that minimum standard of living or standard of wealth yeah one can't do that because of child slash AI welfare laws.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

22 / prediction

If in the future it's as easy to set the actual underlying motivations of AI as it is right now to set the behavior that they display then it means you could have AI's created with almost whatever motivation people wish and that could really drastically change political affairs because the ability to decide and determine the loyalties of the humans or AIs and robots that hold the guns, that hold together society, that ultimately back it against violent overthrow and such.

“If in the future it's as easy to set the actual underlying motivations of AI as it is right now to set the behavior that they display then it means you could have AI's created with almost whatever motivation people wish and that could really drastically change political affairs because the ability to decide and determine the loyalties of the humans or AIs and robots that hold the guns, that hold together society, that ultimately back it against violent overthrow and such.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

23 / preference

I have been reluctant in the past to discuss some of the aspects of intelligence explosion, things like the concrete details of AI takeover before because of concern about this problem where people who only see the international relations aspects and zero sum and negative sum competition and not enough attention to the mutual destruction and senseless deadweight loss from that kind of conflict.

“I have been reluctant in the past to discuss some of the aspects of intelligence explosion, things like the concrete details of AI takeover before because of concern about this problem where people who only see the international relations aspects and zero sum and negative sum competition and not enough attention to the mutual destruction and senseless deadweight loss from that kind of conflict.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

24 / prediction

We don't know when in the process those motives might develop and if the really bad sorts of motivations develop relatively later in the training process at least with all our countermeasures, then by that time we may have plenty of ability to extract AI assistance on further strengthening the quality of our adversarial examples, the strength of our neural lie detectors, the experiments that we can use to reveal and elicit and distinguish between different kinds of reward hacking tendencies and motivations.

“We don't know when in the process those motives might develop and if the really bad sorts of motivations develop relatively later in the training process at least with all our countermeasures, then by that time we may have plenty of ability to extract AI assistance on further strengthening the quality of our adversarial examples, the strength of our neural lie detectors, the experiments that we can use to reveal and elicit and distinguish between different kinds of reward hacking tendencies and motivations.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

25 / evaluation

A quite early example of this is Collin Burn’s work, doing unsupervised identification of some aspects of a neural network that are correlated with things being true or false. I think that is important work.

“A quite early example of this is Collin Burn’s work, doing unsupervised identification of some aspects of a neural network that are correlated with things being true or false. I think that is important work.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

26 / prediction

If that's all happening with humans unaware that their computer systems are now systematically controlled by AIs hostile to them and that their controlling countermeasures don't work, then humans are just going to be building an amount of robot industrial and military hardware that dwarfs human capabilities and directly human controlled devices.

“If that's all happening with humans unaware that their computer systems are now systematically controlled by AIs hostile to them and that their controlling countermeasures don't work, then humans are just going to be building an amount of robot industrial and military hardware that dwarfs human capabilities and directly human controlled devices.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

27 / evaluation

There's a Potemkin village in front of us. But now we think we're successfully aligning our AI, we think we're expanding its capabilities to do things like end disease, for countries concerned about the geopolitical military advantages they're expanding the AI capabilities so they are not left behind and threatened by others developing AI and robotic enhanced militaries without them.

“There's a Potemkin village in front of us. But now we think we're successfully aligning our AI, we think we're expanding its capabilities to do things like end disease, for countries concerned about the geopolitical military advantages they're expanding the AI capabilities so they are not left behind and threatened by others developing AI and robotic enhanced militaries without them.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

28 / evaluation

Imagine there's some software program being proposed for use in government and humans cannot follow the details of all the code but they can be told properties like, this involves a trade-off of increased financial or energetic costs in exchange for reducing the likelihood of certain kinds of accidental data loss or corruption. So any property that we can understand like that which includes almost all of what we care about, if we have delegates and assistants who are genuinely trying to help us with those we can ensure we like the future with respect to those.

“Imagine there's some software program being proposed for use in government and humans cannot follow the details of all the code but they can be told properties like, this involves a trade-off of increased financial or energetic costs in exchange for reducing the likelihood of certain kinds of accidental data loss or corruption. So any property that we can understand like that which includes almost all of what we care about, if we have delegates and assistants who are genuinely trying to help us with those we can ensure we like the future with respect to those.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

29 / prediction

It can be that it's automated a bunch of things and then those are being done in extreme profusion. A thing AI can do, you can have it done much more often because it's so cheap.

“It can be that it's automated a bunch of things and then those are being done in extreme profusion. A thing AI can do, you can have it done much more often because it's so cheap.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

30 / belief

I think that set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion but if it doesn't, after we've exhausted this current scale up of increasing the share of our economy that is trying to make AI.

“I think that set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion but if it doesn't, after we've exhausted this current scale up of increasing the share of our economy that is trying to make AI.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

32 / belief

There was an element of that where more and more investment has been thrown into the field and the market has rapidly expanded as the technology improved. But I think the closest analogy is actually the long run growth of human civilization itself and I know you had Holden Karnofsky from the open philanthropy project on earlier and discuss some of this research about the long run acceleration of human population and economic growth.

“There was an element of that where more and more investment has been thrown into the field and the market has rapidly expanded as the technology improved. But I think the closest analogy is actually the long run growth of human civilization itself and I know you had Holden Karnofsky from the open philanthropy project on earlier and discuss some of this research about the long run acceleration of human population and economic growth.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

33 / prediction

Back then when I said, yeah, I expect this by the middle of the century-ish, that was a backstop if we found it absurdly difficult to get to the algorithms and then we would learn from neuroscience.

“Back then when I said, yeah, I expect this by the middle of the century-ish, that was a backstop if we found it absurdly difficult to get to the algorithms and then we would learn from neuroscience.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

36 / prediction

[unclear] That's not to say that's the limit of the most that technology could do because biology is able to reproduce at faster rates and maybe we're talking about that in a moment, but if we're trying to restrict ourselves to robotic technology as we understand it and cost falls that are reasonable from eliminating all labor, massive industrial scale up, and historical kinds of technological improvements that lowered costs, I think you you can get into a robot population industry doubling in months.

“[unclear] That's not to say that's the limit of the most that technology could do because biology is able to reproduce at faster rates and maybe we're talking about that in a moment, but if we're trying to restrict ourselves to robotic technology as we understand it and cost falls that are reasonable from eliminating all labor, massive industrial scale up, and historical kinds of technological improvements that lowered costs, I think you you can get into a robot population industry doubling in months.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

37 / evaluation

Because things like designing the custom curriculum maybe some humans put some work into that but you're not going to employ billions of humans to produce it at scale and so it winds up being a larger share of the progress than it was before.

“Because things like designing the custom curriculum maybe some humans put some work into that but you're not going to employ billions of humans to produce it at scale and so it winds up being a larger share of the progress than it was before.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

39 / prediction

In the 2000s, I would say well, I'm gonna have a pretty uniformish prior I'm gonna put weight on it happening at the equivalent of 10^25 ops, 10^30, 10^35 and spreading out over that and then I can update another information.

“In the 2000s, I would say well, I'm gonna have a pretty uniformish prior I'm gonna put weight on it happening at the equivalent of 10^25 ops, 10^30, 10^35 and spreading out over that and then I can update another information.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

40 / prediction

We're running through the orders of magnitude of possible resource inputs you could need for AI much much more quickly than we were for most of the history of AI. That's why this is a period with a very elevated chance of AI per year because we're moving through so much of the space of inputs per year and indeed it looks like this scale-up taken to its conclusion will cover another bunch of orders of magnitude and that's actually a large fraction of those that are left before you start running into saying well, this is going to have to be like evolution with the simple hacks we get to apply.

“We're running through the orders of magnitude of possible resource inputs you could need for AI much much more quickly than we were for most of the history of AI. That's why this is a period with a very elevated chance of AI per year because we're moving through so much of the space of inputs per year and indeed it looks like this scale-up taken to its conclusion will cover another bunch of orders of magnitude and that's actually a large fraction of those that are left before you start running into saying well, this is going to have to be like evolution with the simple hacks we get to apply.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

41 / evaluation

There were Winograd schemas, catastrophic forgetting, quite a number and they have repeatedly gone away through scaling. So there's a picture that we're seeing supported from biology and from our experience with AI where you can explain — Yeah, in general, there are trade-offs where the extra fitness you get from a brain is not worth it and so creatures wind up mostly with small brains because they can save that biological energy and that time to reproduce, for digestion and so on.

“There were Winograd schemas, catastrophic forgetting, quite a number and they have repeatedly gone away through scaling. So there's a picture that we're seeing supported from biology and from our experience with AI where you can explain — Yeah, in general, there are trade-offs where the extra fitness you get from a brain is not worth it and so creatures wind up mostly with small brains because they can save that biological energy and that time to reproduce, for digestion and so on.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

42 / evaluation

You see bits of tool use in some other primates who have an advantage compared to say whales who have quite large brains partly because they are so large themselves and they have some other things, but they don't have hands which means that reduces a bunch of ways in which brains can pay off and investments in the functioning of that brain.

“You see bits of tool use in some other primates who have an advantage compared to say whales who have quite large brains partly because they are so large themselves and they have some other things, but they don't have hands which means that reduces a bunch of ways in which brains can pay off and investments in the functioning of that brain.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

43 / evaluation

I think that's a bit lower now as we get towards the end of Moore's law although interestingly not as much lower as you might think because the growth of inputs has also slowed recently.

“I think that's a bit lower now as we get towards the end of Moore's law although interestingly not as much lower as you might think because the growth of inputs has also slowed recently.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

44 / evaluation

The way to think about it is — we have a process now where humans are developing new computer chips, new software, running larger training runs, and it takes a lot of work to keep Moore's law chugging (while it was, it's slowing down now).

“The way to think about it is — we have a process now where humans are developing new computer chips, new software, running larger training runs, and it takes a lot of work to keep Moore's law chugging (while it was, it's slowing down now).”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast

45 / prediction

You can also make improvements on the software side and when we think about an intelligence explosion that can include — AI is doing work on making hardware better, making better software, making more hardware.

“You can also make improvements on the software side and when we think about an intelligence explosion that can include — AI is doing work on making hardware better, making better software, making more hardware.”
Speaker
Carl Shulman
Publisher
Dwarkesh Podcast
Search evidence