How human-like do safe AI motivations need to be?

1. Introduction 

In previous essays, I’ve laid out my rough picture of the path to building increasingly powerful AIs safely – and in particular, to exerting control over their motivations and their options sufficient to allow us to use their labor to massively improve the situation (“AI for AI safety”), especially with respect to our ability to make the next generation of AIs safe. In this essay, I want to address directly a particular sort of concern about this project: namely, that the AIs in question will be too alien to be safe. That is, the thought goes, AIs built/grown via contemporary machine learning methods will end up motivated by a complex tangle of strange, inhuman drives/heuristics that happen to lead to highly-rewarded behavior in training. But in the context of more powerful systems and/or more out-of-distribution inputs, the thought goes, these alien drives will lead to existentially catastrophic behavior. 

This sort of concern is core to certain kinds of arguments for pessimism about AI alignment risk – for example, the argument presented in the recent book “If Anyone Builds It, Everyone Dies,” by Eliezer Yudkowsky and Nate Soares. And I think the concern has real force. However, I also find it less worrying than Yudkowsky and Soares do – especially in AIs with more intermediate levels of capability (that is, the AIs most crucial to “AI for AI safety,” and which I view as the most direct responsibility of human alignment researchers to make safe). 

The core reason for this is that we don’t need to build AI systems with long-term consequentialist motivations we’re happy to see optimized extremely hard. In the context of systems like that: yes, alien motivations are indeed a problem (as are human-like motivations with other flaws, even potentially minor flaws). But systems like that are not the goal. Rather, according to me, the goal is (roughly) to build AI systems that follow our instructions in safe ways. And this project, in my opinion, admits of a much greater degree of “error tolerance.” 

In particular: the motivations that matter most for safe instruction-following are not the AI’s long-term consequentialist motivations (indeed, if possible, I think we mostly want to avoid our AIs having this kind of motivation except insofar as it is implied by safe instruction-following). Rather, the motivations that matter most are the motivations to reject options for rogue behavior – that is, motivations that are applied centrally to actions rather than long-term outcomes. Or to put it another way: a lot of the safety we’re getting via motivation control is going to rest on AIs being something more akin to “virtuous” or “deontological” with respect to options for rogue behavior, rather than from AIs directly caring about the same long-term outcomes as we do. And the relevant form of virtue/deontology, in my opinion, need not be fully or even especially human-like in the concepts/drives/motivations that structure it.¹ Rather, it just needs to add up, in practice, to safe behavior on any dangerous inputs (that is, inputs that make options for rogue behavior available) that the AI is in fact exposed to. 

Admittedly: this reply doesn’t address all of the standard concerns about relying on non-consequentialist motivations for safety – for example, concerns about AIs with at least some long-term consequentialist motivation going rogue via the “nearest unblocked strategy” that is suitably compatible with the non-consequentialist considerations they care about. Nor does it provide additional comfort with respect to preventing alignment faking, or with respect to what I’ve previously called the “science of non-adversarial generalization” – that is, the challenge of ensuring (on the first safety-critical try) that the motivations of non-alignment-faking AIs generalize safely to practically-relevant out-of-distribution inputs. To the contrary, I do think that AI motivations being less human-like makes these challenges harder, because the alien-ness at stake makes it harder for humans to predict how the motivations will apply in a given case. 

But this, I think, is an importantly different concern than the one at stake in the central argument of “If Anyone Builds It, Everyone Dies” – and one, I think, about which existing levels of success at alignment in current systems (together with: existing success at out-of-distribution generalization in ML more generally) provides greater comfort. That is: to the extent that the degree of good/safe generalization we currently get from our AI systems arises from a complex tangle of strange alien drives, it seems to me plausible that we can get similar/better levels of good/safe generalization via complex tangles of strange alien drives in more powerful systems as well. Or to put the point another way: currently, it looks like image models classify images in somewhat non-human-like ways – e.g., they’re vulnerable to adversarial examples that humans wouldn’t be vulnerable to. But this doesn’t mean that they’re not adequately reliable for real-world use, even outside the training distribution. Aligning AIs with alien motivations might, I think, be similar. 

All that said: at a higher level, relying on smarter-than-human AIs with strange alien drives to reject options to seek power/control over humans seems extremely dangerous. I am more optimistic than Yudkowsky and Soares that it might work; but I share their alarm at the idea that we would need to try it. And to the extent we end up needing to try it with earlier generations of AIs, I think a key goal should be to transition rapidly to a different regime. 

I’ll be starting a job at Anthropic soon, but I’m here speaking only for myself, and Anthropic comms hasn’t reviewed this post. Thanks especially to Nate Soares and Holden Karnofsky for extensive discussion of some of these issues.

2. Are our AIs like aliens?

Let’s start by laying out the concern about alien AIs in a bit more detail, focusing on the presentation in “If Anyone Builds It, Everyone Dies” (IABIED). 

We can understand the core argument in IABIED roughly as follows: 

1. AIs built via anything like current techniques will end up motivated by a complex tangle of strange alien drives that happened to lead to highly-rewarded behavior during training. 
2. AIs with this kind of motivational profile will be such that “what they most want” is a world that is basically valueless according to humans. 
3. Superintelligent AIs with this kind of motivational profile will be in a position to get “what they most want,” because they will be in a position to take over the world and then optimize hard for their values.
4. So, if we build superintelligent AIs via anything like current techniques, they will take over the world and then optimize hard for their values in a way that leads to a world that is basically valueless according to humans. 

We can query various aspects of this argument – and I won’t try to evaluate all of it in detail here. For now, though, let’s focus on the first premise. Is that right? 

I’m not sure it is. Notably, for example: current AI pre-training focuses their initial development specifically on a vast amount of human content, thereby plausibly endowing them with many quite human-like representations. That is: current AIs need to understand at a very early stage what human concepts like “helpfulness,” “harmlessness,” and “honesty” mean. And while it is of course possible to know what these concepts mean without being motivated by them (cf “the genie knows but doesn’t care”), the presence of this level of human-like conceptual understanding at such an early stage of development makes it more likely that these human-like concepts end up structuring AI motivations as well. 

Indeed, this is one of many notable disanalogies between AI training and natural selection – one of Yudkowsky and Soares’s favorite reference points. That is, pointing human motivations directly at something like “inclusive genetic fitness” wasn’t even an option for natural selection, because humans didn’t have a concept of inclusive genetic fitness until quite recently. But AIs will plausibly have concepts like “helpfulness,” “harmlessness,” “honesty” much earlier in the process that leads to their final form. 

What’s more, existing efforts at interpretability do in fact often uncover notably human-legible representations at work in current AI systems (though obviously, there are serious selection effects at stake in this evidence);² and to the extent such representations correspond to important/natural “joints in nature” generally useful to understanding the world, this is all the less surprising. And to my mind, at least, the ease with which we’ve been able to prompt quite human-like and aligned behavior in our AIs using quite basic, RLHF-like techniques is both notable and, in my opinion, in tension with the naive predictions of a worldview that treats AI cognition as extremely alien. In particular: in my opinion, we haven’t just succeeded at getting fairly reliably aligned behavior on a specific training distribution. Rather, we’ve succeeded at creating dispositions towards aligned behavior that generalize fairly (though of course, not perfectly) well to new, at-least-somewhat out of distribution inputs as well – and success at this kind of generalization is effectively what “human-like-ness” consists in.³ (Here I expect Yudkowsky and Soares will say that the sort of generalization we care about most is importantly different; I’ll address this concern later in the essay.)

Of course, it’s true that current AIs do sometimes behave quite badly – including, sometimes, in quite alien ways. But in interpreting this kind of evidence, my sense is that people worried about AI alignment often try to have the evidence both ways. That is, they treat incidents like Bing Sydney as evidence that alignment is hard, but they don’t treat the absence of more of such incidents as evidence that alignment is easy; they treat incidents of weird/bad out-of-distribution AI behavior as evidence alignment is hard, but they don’t treat incidents of good out-of-distribution AI behavior as evidence alignment is easy. Of course, you can claim to learn nothing from any of these data points, and to be using them only to illustrate your perspective to others. But if you take yourself to be learning from the bad cases, I think you should be learning from the good cases, too.⁴ 

Indeed, my sense is that many observers of AI have indeed taken a lot of comfort from the good cases. That is, the intuition goes: if AI alignment remains effectively this easy going forwards, then things are looking pretty good. And while I generally think that casual comfort of this kind is notably premature, I share some intuition in the vicinity. In particular, I have some hope that by the time we start building AIs that can be transformatively useful – e.g., AIs within the “AI for AI safety sweet spot” that I discussed earlier in the series – alignment has not become radically harder, and in particular, that efforts to ensure safe instruction-following behavior continue to generalize out-of-distribution at least as well as they have done so far. And I think it plausible that if transformatively useful AI systems safely follow instructions about as reliably as current models do (and especially: if we can get better at dealing with reward-hacking-like problems that might mess up capability elicitation), this is enough to safely elicit a ton of transformatively useful AI labor, including on alignment research – and that the game-board looks substantially better after that.

What’s more, while I am indeed concerned about the many incidents of bad/misaligned behavior in current models, I don’t think any of these currently look like full-blown incidents of the threat model made most salient by concerns about AI alien-ness in particular. In particular, while it’s true that we see AIs willing to engage in problematic forms of power-seeking – e.g., deceiving humans, self-exfiltrating, sandbagging, resisting shut-down, etc – they currently mostly do so in pursuit either of fairly human-legible or context/prompt-legible goals like helpfulness or harmlessness (e.g. here and here); on the basis of shifting between different human-legible personas (e.g. here and here); in pursuit of completing the task itself (e.g. here and here); or, perhaps, in pursuit of “terminalized” instrumental goals like an intrinsic drive towards power/survival (this is another interpretation of some of the results previously cited). That is: in my opinion, we have yet to see especially worrying cases of AIs going rogue specifically in pursuit of goals (and especially, long-term consequentialist goals) that seem especially strange/alien – though of course, this could change fast. 

What’s more, in thinking about what it would mean for an AI’s motivations/cognition to be human-like or alien, I think we need to be careful about the level of abstraction at which we are understanding the claim in question. That is: it’s not enough to say that AI behavior emerges from a complex tangle of heuristics, circuits, etc rather than something more cleanly factored (since: the same is true of human behavior); nor, more importantly, to say that the heuristics/circuits/etc work in a different way in the AI case. Rather, we should be focused on the high-level behavioral profile that emerges in the system as a whole, and the degree to which it diverges from some more human-like alternative.⁵ And as I’ll discuss below, what actually matters is whether it diverges in practice, and in catastrophic ways – not just whether it does so in some way on some inputs. Thus, per the analogy I discussed in the intro: an AI classifying cat pictures adequately doesn’t actually need to mimic human judgments across every single case (nor, indeed, will human judgments all agree with one another); to be robust to every adversarial example; etc. Rather: it just needs to get enough cases (including: enough cases out of distribution) enough right. And in this respect, for example, it looks to me like many of our efforts to get AIs to behave in fairly human-like ways, including out of distribution, are going OK. 

All that said: I remain sympathetic to some versions of premise (1). In particular: I think it quite plausible that if we really understood how current AI systems think/make decisions etc, we would indeed see that to the extent they are well-understood as having motivations at all, these motivations are quite strange/alien indeed, and that they will indeed lead to notably alien behavior on a wide variety of realistic inputs (beyond what we’ve seen thus far). For example, while I think it’s an open question exactly how to interpret data of this kind (see e.g. discussion here), I definitely get some (fairly creepy) vibe of “strange, alien mind” from e.g. the sorts of chains of thought documented in this work on scheming from Apollo and OpenAI (full transcripts here):

And I think it plausible, as well, that this sort of alien-ness will persist and/or increase in more powerful AIs built using similar techniques – and this even accounting for moderate improvements in our behavioral science and transparency tools. 

That is: overall, I share Yudkowsky and Soares’ concern that the motivations of AIs built using current techniques will remain importantly strange/alien. And so I want to examine in more detail the implications for safety if this is true. In particular: if, in fact, powerful AI systems are motivated in at-least-somewhat alien ways, does that mean we are as doomed as Yudkowsky and Soares think? 

I’m skeptical. In particular: I think the Yudkowsky/Soares argument above places too much emphasis on long-term consequentialist AI motivations in particular, and that it neglects the ways in which the sort of safety accessible via shorter-term and especially non-consequentialist motivations ends up more tolerant of error. Or to put the point in more Yudkowskian terminology, I think that something like “corrigible AIs” (that is, roughly, AIs with imperfect motivations but which nevertheless obey your instructions and don’t go rogue) can safely be more notably alien (and otherwise flawed) in their motivations than “sovereign AIs” (that is, AIs with motivations so perfect you trust them to optimize the future arbitrarily hard without accepting any of correction from you) – and that our focus should be on something like corrigibility in particular.⁶ Let me say more about what I mean. 

3. Value fragility, sovereign AIs, and getting AI motivations “exactly right”

In my opinion, we should understand the concern about “alien AIs,” and its role in the argument I laid out above, as a specific version of a broader concern that has haunted the AI alignment discourse from basically the beginning: namely, the concern that safe AI motivations need to be, in some sense, “exactly right.” That is, the more general argument in the background (though: not stated as explicitly in IABIED) is something like the following (alterations from the previous version in bold):⁷ 

1. AIs built via anything like current techniques will end up with motivations that aren’t exactly right. 
2. AIs whose motivations aren’t exactly right will be such that “what they most want” is a world that is basically valueless according to humans. 
3. Superintelligent AIs with this kind of motivational profile will be in a position to get “what they most want,” because they will be in a position to take over the world and then optimize hard for their values.
4. So, if we build superintelligent AIs, they will take over the world and then optimize hard for their values in a way that leads to a world that is basically valueless according to humans. 

Why think that AI motivations need to be exactly right? Well, roughly, the basic concern is that human value is “fragile” under extreme optimization. That is, the thought goes: extreme optimization for slightly-flawed values leads to places that are basically valueless by human lights; and superintelligences will be forces for extreme optimization. 

I’ve written in some detail, elsewhere, about my takes on concerns about “value fragility” of this kind. See, in particular, this set of informal notes, and this longer essay about whether the concern in question applies similarly to humans with respect to one another. For those interested, I’ve also given a summary of some of those takes in an appendix below. 

However, while I think there are a variety of interesting and important questions we can raise about value fragility (and especially: about the extent to which similar concerns do or do not apply between different humans), I’m not, here, going to dispute a certain kind of broad concern about it. That is, I’m going to accept that for long-term consequentialist value-systems that aren’t exactly right (or at least, which don’t put non-trivial weight on something exactly right), if you optimize for them super hard, you do indeed create a world that is roughly valueless by human lights. And I’m going to accept, further, that the degree of alien-ness at stake in the motivations of AIs developed via current techniques is likely enough to fall short of “exactly right” in this sense (at least to the extent that such AIs develop long term consequentialist motivations at all – something which, as I’ll discuss below, I think we should be trying to prevent except insofar as these motivations follow from safe instruction-following).

What follows from this? Basically: what follows is that current techniques aren’t fit to build AI systems with long-term consequentialist motivations that we’re happy to see optimized extremely hard. That is, roughly, they are not fit for building what Yudkowsky call a “Sovereign AI” – that is, in his words, an AI that “wants exactly what we extrapolated-want and is therefore safe to let optimize all the future galaxies without it accepting any human input trying to stop it.”

But building sovereign AIs of this kind, I claim, should not be our goal. Indeed, I explicitly defined solving the alignment problem so as to neither require this degree of alignment in the AIs we build; nor, even, to require the ability to elicit the creation of AIs that are this degree of aligned (and in particular, I’m not counting “build an AI that you’re happy to make dictator-of-the-universe” as one of the “main benefits of superintelligence”). This is centrally because I think that building an AI worthy of this degree of trust may be a notably more difficult challenge than building an AI that safely follows our instructions.⁸ But also, even aside from the technical difficulty of building a “sovereign AI” of this kind, I don’t think we should view “now we’ve handed control of the world to a perfectly benevolent AI dictator/oligarchy” as a clearly ideal end state of our efforts on alignment – nor, indeed, one that is unavoidable absent some other sort of enforced restriction on AI development.⁹ To the contrary, I think we should focus more on a vision of humans who are able to get safe, fully-elicited superintelligent help in navigating the ongoing transition to even greater levels of AI capability – including with respect to questions about what sorts of “sovereign” to make what sorts of AIs going forwards.¹⁰ 

That said: the argument for pessimism at stake in IABIED – and also, in the more general value-fragility argument outlined above – isn’t “we should aim for perfect AI dictators, but we’re going to get alien/imperfect AI dictators instead.” Rather, it’s more like: “we’re not going to be able to avoid getting AI dictators of some form, and the dictators we’re going to get will be alien/imperfect.” That is: Yudkowsky and Soares do recognize the possibility of trying to build what Yudkowsky calls “corrigible AI” – that is, in Yudkowsky’s words, an AI “which doesn’t want exactly what we want, and yet somehow fails to kill us and take over the galaxies despite that being a convergent incentive there.” And indeed, my understanding is that Yudkowsky and Soares agree with me, as a first pass, that “corrigible AI” in this sense is a better near-term focus of efforts at alignment. But they think that the project of building corrigible AI, too, is doomed to fail. 

Now, as I’ve discussed in some informal notes elsewhere, I think that the role of the notion of “corrigibility” in the discourse about AI alignment is often unclear/ambiguous. In particular, in the context of the Yudkowsky quote above, it basically just means “any powerful AI with not-exactly-right values that is somehow otherwise safe.” But often, people use the term to refer to a number of more specific properties – notably, a willingness to submit to corrective intervention like shut-down or values-modification (while remaining useful in other ways – e.g. not trying to shut itself down). Conceptually, these aren’t actually the same. For example: humans don’t perfectly share each other’s values, and they will generally resist “corrective intervention” like shut-down (death) and values-modification (brainwashing), but they also aren’t at least presently aiming at omnicide.

Still, something like “corrigibility” is indeed a closer match for my focus in this series than “sovereign AI.” That is: I want us to learn how to build AIs that safely follow our instructions – where “safely” means “without engaging in rogue behavior,” and where “rogue behavior” includes things like resisting shut down/values-modification, and definitely includes taking over the world. Indeed, I think it’s notable that on the most straightforward understanding of plans for building “sovereign AI,” the AI in question does take over the world (for example, the classic argument for AI takeover I laid out here applies with roughly comparable weight to AIs with perfect long-term consequentialist values)  – it’s just that, what it does from there is suitably valuable by human lights.¹¹ That is, in a sense, even sovereign AIs with exactly-right long-term values go “rogue” in my sense – it’s just that, after humans lose control, the future is still good. But I’m interested in avoiding rogue behavior period. 

4. What does it take to build corrigible AIs?

What does that involve? Well, let’s return, for a moment, to the framework for thinking about AI incentives that I laid out in the second essay in the series. 

On this framework, the central aspects of an AI’s motivation profile that matter are its satisfaction (how happy it is with the best non-rogue option), its inhibition (how averse it is to various paths to successful rogue behavior), its failure aversion (how much it wants to avoid the downsides of failed rogue behavior) and its ambition (how much it wants the power at stake in rogue behavior). The reason AIs with “alien” drives take over the world is that what they most want in the long term makes them suitably “ambitious” about getting the power at stake. And as I noted above, I actually think that similar considerations would suggest that even AIs with exactly-right long-term consequentialist values would take over as well. 

Suppose, though, that we don’t want our AIs to take over or to go rogue at all. How can we ensure this? A big part of it is option control – and in particular, restricting the options for rogue behavior available to the AIs in question. Indeed, as I discussed in the second essay, I think the classic argument for AI risk is best understood as relying heavily on the premise that rogue-option-restriction of this kind will be so ineffective that the AI in question will be in a position to take over the world extremely easily (such that “failure aversion” isn’t an important part of the story), and to do so via such a wide variety of methods that it can successfully route around any deontology-like inhibitions that make certain routes undesirable (cf concerns about “nearest unblocked neighbor”). In this context, that is, the main thing that matters for whether the AI has an incentive to take over is whether it has any suitably long-term consequentialist motivation that gives rise to even a small amount of ambition. And indeed, as I’ll discuss below, I think we see this kind of low/weak standard at work in the argument in IABIED for expecting AIs with alien drives to have long-term consequentialist drives in particular. 

As I discussed in my last essay, though, I don’t think we should give up on option control playing an important role in the safety of advanced AIs, nor do I think that success in this respect needs to be all or nothing (i.e., no viable options for rogue behavior vs. can take over the world extremely easily via tons of different methods). And in the context of understanding “corrigibility,” I think the availability of option control as a possible tool is important. In particular, to the extent you are hoping to rule out a suitable range of paths to rogue behavior somehow, option control allows you to do so via intervention on the AI’s environment/capabilities as well as via its motivations. In this sense, as with the notion of “alignment” more broadly, “corrigibility” in the sense I care about is importantly relative to a particular environment and capability level. That is, the AI in question doesn’t need to act corrigibly across all possible inputs and capability levels – it just needs to act corrigibly in the specific context you care about, on the specific set of tasks you’re trying to get it to perform. 

Beyond option control, though, we can divide the motivational aspect of corrigibility into two components: 

1. Minimizing the AI’s “ambition.”
2. Ensuring that the other aspects of the AI’s motivational profile (its satisfaction, inhibition, and failure aversion) are sufficiently strong/robust as to outweigh the degree of ambition it does have. 

Let’s look at each in turn. 

5. IABIED’s “alien motivations” argument isn’t about corrigibility

I think that all of the issues I just listed are indeed problems for crafting corrigible AI agents that nevertheless optimize tenaciously (but safely) for long-term consequentialist outcomes when instructed to do so. But I think that these problems are importantly different from the central problem with human-like vs. alien motivations at stake in IABIED. That is: the central problem at stake in IABIED is that because AIs have alien motivations, their favorite long-term consequentialist outcome isn’t valuable by human lights. But in the context of corrigibility, the question isn’t whether an AI’s favorite long-term consequentialist outcome is valuable by human lights. Rather, the question is whether the AI is suitably motivated to reject the sort of problematic power-seeking that optimizing for any long-term consequentialist outcome – whether good or bad by human lights – tends to incentivize. 

Do alien motivations make that problem harder as well? I think: yes. But unlike “pointed at exactly my values-on-reflection,” I think corrigibility is actually compatible with AIs having somewhat non-human-like motivations, in the same way that suitably accurate cat-classification is compatible with AIs having somewhat non-human-like conceptions of cats (such that e.g. they are vulnerable to adversarial examples that humans aren’t). Let me say more about what I mean.

6. What difference does human-like-ness make?

To better hone in onwhere human-likeness makes what sort of difference to corrigibility, recall the four-step framework I laid out in “Giving AIs safe motivations,” namely: 

1. Instruction-following on safe inputs: Ensure that your AI follows instructions on safe inputs (i.e., cases where successful rogue behavior isn’t a genuine option), using accurate evaluations of whether it’s doing so.
2. No alignment faking: Make sure it isn’t faking alignment on these inputs – i.e., adversarially messing with your evidence about how it will generalize to dangerous inputs.
3. Science of non-adversarial generalization: Study AI generalization on safe inputs in a ton of depth, until you can control it well enough to be rightly confident that your AI will generalize its instruction-following to the dangerous inputs it will in fact get exposed to.
4. Good instructions: On these dangerous inputs, make it the case that your instructions rule out the relevant forms of rogue behavior.

I think that the concern about alien motivations (at least: as present in IABIED) is best understood as a concern about steps 2 and 3. That is: step 1 is centrally about getting a certain kind of behavior on the training distribution (and on other safe inputs), whereas the “alien motivations” concern is framed as centrally one of generalization to dangerous inputs (e.g., “you don’t get what you train for”). So let’s assume, going forwards, that we’ve completed step 1 successfully. 

And step 4, notably, assumes that you are able to structure the AI’s motivations using the human-like concepts at stake in the instructions. Indeed, in a sense, the difficulty of step 4 provides a useful baseline for thinking about the difficulty of alignment in the context of human-like motivations – not because humans are motivated to follow instructions, but because the instructions are given in human-like terms, and will (by hypothesis) be interpreted in human-like ways. 

Now, notably: all of the corrigibility issues I described above apply even to instruction-following AIs. That is: to the extent we want to be able to instruct those AIs to do things like “make me lots of money over the next ten years,” we also need ways to include suitably robust constraints/exclusions like “but don’t resist shut-down, don’t try to prevent me from changing these instructions even though this would lead to me having less money in ten years, don’t create successor agents that won’t follow these instructions, etc”; to give them the right amount of weight in the AI’s overall motivational profile; and so on. 

So one question we can ask is: how hard is step 4? But as I discussed in “Giving AIs safe motivations,” I am decently optimistic in this respect. This is partly because I think that many of the most important forms of rogue behavior may be reasonably easy to identify and rule out ahead of time; partly because I think we might be in a position to point more directly at deeper generators of our intuitions about what corrigible behavior looks like in a given case (in the limit, for example, you might be able to instruct the AI to behave “corrigibly”²⁷); and partly because I think that if we actually make it to step 4 in this way, we’ll be able to draw on a ton of help from other instruction-following AIs in red-teaming and improving our instructions. For present purposes, though, the difficulty of step 4 doesn’t matter, because the “alien motivations” problem is supposed to bite at steps 2 and 3. So let’s assume that we’ve completed step 4 successfully as well. That is, we have instructions available such that if our AIs are instruction-following on the practically relevant dangerous inputs, they’ll be corrigible. 

OK, so what about steps 2 and 3? These steps are about ensuring that the AI’s instruction-following generalizes from safe inputs to dangerous inputs – where step 2 is about ruling out scenarios where the AI’s good behavior on safe inputs is actively calculated to mislead you about how it will generalize, and step 3 is about ensuring good generalization in the absence of this kind of adversarial dynamic. 

Now suppose that we hypothesize, per the argument above about alien motivations, that any good behavior you successfully get in the context of step 1 emerges from a complex tangle of alien drives and heuristics, which happen to lead to desired (in this case: instruction-following) behavior during training, but which will lead to quite alien behavior in some other circumstances. How much of a problem is that? 

Well, if we were assuming that our AI’s motivations need to be exactly right, then it would be a very big problem. But we’re not assuming that. Rather, what we need in the present context is for either of the following to be true: 

1. Either these alien motivations nevertheless give rise to instruction-following behavior on the specific set of dangerous inputs we care about, OR
2. To the extent these alien motivations lead to something other than instruction-following behavior on those dangerous inputs, this behavior is nevertheless not catastrophically dangerous (e.g., the AI starts acting very weirdly, but it doesn’t specifically start seeking power in problematic ways). 

But now the argumentative gap between “alien motivations” and “will go catastrophically rogue on the dangerous inputs we care about” becomes quite clear. That is: alien motivations means that you get some kind of weird behavior on some hypothetical inputs. But it doesn’t, yet, mean that you get catastrophically dangerous power-seeking on the specific, practically-relevant inputs we care about. 

In my opinion, one of the biggest problems with the argument in IABIED – both in the book, and in the online supplementary materials – is that it does not do enough to bridge this argumentative gap. That is: it seems to me that the discussion is pervaded by the assumption that an advanced AI’s motivations need to be “exactly right,” and that its treatment of the specific sort of generalization we need is correspondingly under-developed. In particular, I think the book is too frequently satisfied, effectively, with arguing that “this AI’s motivations will not generalize perfectly across all scenarios”; and that the concern about alien motivations, as articulated in the book, mostly amounts to a restatement of this thesis. 

Or to put the point another way: I think the book mostly lacks any serious engagement with the question of when ML training can and cannot ensure adequately good generalization off of the training distribution. Indeed, it sometimes seems to me that a lot of Yudkowsky and Soares’s concern with machine learning amounts to a fully general concern like: “ML can’t ensure good OOD generalization, because there are too many functions that fit the limited data you provide.”²⁸ But I think concerns at this level of generality fail to account for the degree of good generalization we do in fact see in ML systems; and they fail, too, to look in adequate detail at the specific degree of good generalization we need. 

6.3 Out of distribution cats vs. maximal cats

In general, I suspect that robustness to small differences/degradations in motivations is generally quite a bit easier to achieve in the context of the more deontological/virtue-ethical motivations that I’ve suggested are paradigmatic of corrigibility, relative to the sort of long-term consequentialist motivations at stake in attempting to build a “sovereign AI.” That is: if I trust Sally in some manner X, and the question is whether to trust Bob in a similar way, I am generally more worried about small differences between Bob and Sally’s motivations when X is something like “optimize the universe for exactly the right values” rather than “reject options to take over the world.” 

Why is this? Basically: my intuition is that the values at stake in deontology/virtue-ethics/corrigibility aren’t subject to “optimization” and “maximization” in the same way that the values at stake in long-term consequentialism are. Thus, for example, consider the contrast between trying to classify out-of-distribution cat images correctly, and trying to create the maximally cat-like image. I have the intuition that success at the former task generally requires a lower standard of similarity to the correct concept of “cat” than success at the latter, because the concept, in the former case, isn’t directly subject to optimization pressure in the same way. 

There’s also a separate question of how to best test current AI answers to questions like “what is the maximally cat-like image”  – e.g., whether you should try prompts to this effect, which currently yield quite reasonable answers (see images below), or whether you should try to use e.g. gradient methods to search for inputs that yield the highest probability of being classified as a cat by a network. I generally lean towards something more like the latter method (analogy: humans don’t necessarily know what tastes best to them), but I think it’s conceptually a bit unclear. And it’s unclear, also, what sorts of results we’d get from applying similar gradient methods to humans – and how intuitively “human-like” they would seem.

Regardless of how we test what happens when AIs target their concepts with direct maximization, though, it seems to me that the broad structure of the deontology/virtue-ethics/corrigibility we want out of AIs looks more like “correctly classify out of distribution inputs” than “create concepts robust to being directly maximized.” That is, what we want, centrally, out of deontology/virtue-ethics/corrigibility is for the AI’s decision-criteria to pick out the actions that violate the instructions (and/or, which involve going rogue more generally) as not-to-be-done, even in out-of-distribution settings. And this looks to me more like a classification task than a task that involves maximization/optimization of the concepts in question. 

7. Is there something special about the safe-to-dangerous leap that makes alien motivations problematic?

In the previous section, I mostly just argued that alien motivations are compatible with some degree of good out-of-distribution generalization, in the same way that image classifiers can correctly classify out of distribution photos despite using fairly non-human-like forms of cognition. It’s possible, though, to concede this point, but nevertheless to argue that the specific sort of OOD generalization at stake in the leap from safe inputs to dangerous inputs ((i.e., from AIs having no viable options for rogue behavior to AIs having viable options of this form) is such that we should expect catastrophically dangerous generalization failure in this context in particular (at least if the AIs have alien motivations of the form I’ve been focusing on). Let’s look at some considerations in this respect in more detail.

8. Alien AIs are still extremely scary

Overall, then, even if “AIs trained with current methods will have alien motivations as a strong default” is true (and I do think it’s plausible), I don’t think it’s enough to doom alignment on its own. And I think this, especially, in the context of AIs at an intermediate level of capability – AIs, that is, where I think we have much stronger prospects of effectively restricting (even if not, fully eliminating) the rogue options they have available (and hence, where the standards our efforts at motivation control need to meet are lower), and where I expect we’ll be in a better position to effectively evaluate/oversee their task-performance. Indeed, as I mentioned above, one of my central hopes for alignment is that AIs at this kind of intermediate level of capability will be roughly as instruction-following as current AIs are (e.g., even if they act weirdly or even go rogue in some cases, they aren’t consistently scheming and are mostly following instructions in common-sensical ways) and that this level of alignment will be enough to elicit the sort of transformatively useful AI labor at stake in my discussion of “AI for AI safety.” 

All that said, though: I want to be clear that, at a higher level, if the motivations of ML-based AI systems are indeed alien in this way as a strong default, this is an extremely scary situation. That is: this means that as a strong default, ML-based methods of developing advanced AI will involve relying on powerful artificial agents motivated by strange tangles of alien, not-well-understood drives to safely and reliably follow our instructions regardless. Per my discussion above, I think it’s possible that we can make this work. But we should be trying extremely hard not to bet on it. 

Of course, “alien by strong default” is different from “unavoidably alien,” and we can imagine scenarios in which our understanding and control over the motivations ML-based AI systems develop reaches a point where we can ensure a much greater degree of human-like-ness. Indeed, absent alignment faking and evaluation problems, “alienness” in the sense at stake here is centrally, just, a failure of non-adversarial generalization – and we are, at least, in a position to study the dynamics at stake in non-adversarial generalization in a ton of detail. At the least, then, if ML-based AI systems are alien in this sense as a strong default, suitable efforts at red-teaming should generally be reasonably effective at revealing the alien-ness in question (analogy: suitable efforts at red-teaming would’ve been able to show quite clearly that non-scheming humans in the ancestral environment did not intrinsically value reproductive fitness); and if we can avoid compromising the signal that this kind of red-teaming provides (e.g., we continue use some versions of it as validation rather than as a part of training), it (together with other techniques – e.g., transparency tools) might help us iterate towards much more human-like patterns of generalization. And as I’ve tried to emphasize above, this kind of alien-ness comes in importantly varying degrees, depending on the specific size and type of distributional shift that preserves desired/human-like forms of generalization. 

If we can’t avoid alien-ness by improving our understanding/control over the patterns of generalization that ML-based training creates, though, then by the time we’re building superintelligence, we will need to either figure out a way to handle the risks of alien-ness even in the context of full-blown superintelligences, or find some way to transition to a different and more precisely controllable paradigm of AI development (e.g., the “new paradigm” I discussed in the context of transparency tools). And especially in the context of the later stages of the path to safe superintelligence – the stages that I think we should be centrally focused on getting advanced AI help with, rather than attempting ourselves – I think we should be putting a lot of effort towards this latter option. That is: despite all my comments in this essay, I still think we really want to avoid having to build full-blown superintelligences that are motivated by strange tangles of alien drives. So a key goal for automated alignment research should be to give us other options. 

9. Next up: building AIs that do human-like philosophy

OK: that was a long discussion of the concern that AI motivations will be too alien to be safe. In the next essay, I’m going to turn to a different but related concern: namely, that on top of more standard forms of scientific progress, success at AI alignment will require an unrealistic amount of philosophical progress as well. 

Appendix 1: On value fragility

This appendix summarizes some of my current takes on “the fragility of value.” See here, here, and here for more detailed discussion.

• I think that the standard sorts of theoretical justifications for value fragility (e.g., appeals to “extremal goodhardt”³⁷) also predict that even slight psychological differences between humans, and within a single human-over-time, would lead to the sort of problematic divergence in question (and that actual observed differences in human value systems, together with observed degrees of selfishness/indexicality in human values, raise further questions in this respect). This isn’t necessarily an objection to the role of value fragility in the discourse about AI alignment. But to the extent the concept makes implausible/counterintuitive predictions about humans, we should wonder about what’s specifically different in the AI case (though, there is indeed an answer here: namely that AIs are much more psychologically different from humans than humans are from one another). And to the extent it makes worrying predictions about humans, we should extend our worry accordingly.³⁸
• I have yet to see the theoretical justifications for value fragility worked out rigorously; I think vague gestures like Stuart Russell’s here and Yudkowsky and Soares’s here (see their footnote on linear functions and convex polygonal regions) are notably insufficient; and I think greater precision about the scope and dynamics in play would be useful in understanding where and when to expect them to apply.
• Most discussions of value fragility assume unipolar optimization that goes unchecked by other actors, and many people have the intuition that a greater “balance of power” helps to mitigate some of the problem. I don’t yet see a very strong story about why this would be; but I still wonder whether there might be something there. 
• To the extent we expect different humans to converge in their values given some kind of idealized reflection, we should be very interested in what it would take for AIs to fall within a similar basin – and I don’t actually think it’s clear that current AI personas are all that far from the relevant human distribution in this respect. Indeed, there is a case to be made that current AI personas are or could be relatively easily made to be notably closer to certain ideals of human virtue and reflectiveness than many humans are. 
• There are a number of relatively clear cases in which value fragility concerns don’t apply – for example, downside-focused views that mostly want to avoid the presence of certain specific states of affairs (e.g., suffering), and “nice” value systems that care intrinsically at least somewhat about how things go by the lights of other value systems that they might otherwise be “fragile” with respect to. And while I’m sympathetic to concerns that “niceness” of this form is also a fairly narrow target, that actually-good types of niceness themselves need to be something like “exactly right” if they’re going to be optimized hard, and that human forms of “niceness” likely emerged via contingent processes that won’t apply to AIs by default, I think there are some interesting questions about whether the niceness humans display might be sufficiently non-contingent as to provide some comfort (see e.g. the debate here). 
• The broader discourse about value fragility has been influenced specifically by models of rational agents maximizing utility functions (and typically: utility functions understood in impartial and consequentialist terms), and for this reason, I worry that it will end up smuggling in confusions related to things like: an overly reified conception of something like an agent’s “coherent extrapolated volition” or “values-on-reflection” (in particular, a conception that assumes that objects like these exist, are suitably determinate in their content, and provide the governing standards to which all smart agents attempt to conform); wrongful assumptions that smart and effective agents must be well-understood as maximizing a coherent utility function over universe histories that remains consistent over time, rather than satisfying more minimal criteria (e.g., consistent ranking of options at a given time); and some more general over-anchoring on rational-agent-like models.