Joseph Carlsmith RSS Feed

Video and transcript of talk on writing AI constitutions

Thu, 09 Apr 2026 17:02:18 GMT

From a talk at Yale Law School in March 2026.

On restraining AI development for the sake of safety

Thu, 19 Mar 2026 16:18:26 GMT

My take on slowing down AI.

Building AIs that do human-like philosophy

Thu, 29 Jan 2026 17:44:11 GMT

AIs will face philosophical questions humans can’t answer for them.

Video and transcript of talk on human-like-ness in AI safety

Wed, 17 Dec 2025 07:17:15 GMT

From a talk I gave at Constellation in December 2025.

How human-like do safe AI motivations need to be?

Wed, 12 Nov 2025 05:13:55 GMT

AIs with alien motivations can still follow instructions safely on the inputs that matter.

Leaving Open Philanthropy, going to Anthropic

Mon, 03 Nov 2025 17:21:39 GMT

On a career move, and on AI-safety-focused people working at AI companies.

Controlling the options AIs can pursue

Mon, 29 Sep 2025 17:01:59 GMT

On blocking paths to power, and on making deals.

Video and transcript of talk on giving AIs safe motivations

Sun, 21 Sep 2025 23:41:28 GMT

From a talk at UT Austin in September 2025.

Giving AIs safe motivations

Mon, 18 Aug 2025 08:08:08 GMT

A four-part picture.

Video and transcript of talk on “Can goodness compete?”

Thu, 17 Jul 2025 17:17:12 GMT

From a public talk on long-term equilibria post-AGI, given at Mox in SF in July 2025.

Video and transcript of talk on AI welfare

Thu, 22 May 2025 16:10:42 GMT

An overview of my take on AI welfare as of May 2025, from a talk I gave at Anthropic.

The stakes of AI moral status

Wed, 21 May 2025 17:21:45 GMT

On seeing and not seeing souls.

Video and transcript of talk on automating alignment research

Wed, 30 Apr 2025 07:17:04 GMT

From a talk at Anthropic in April 2025.

Can we safely automate alignment research?

Wed, 30 Apr 2025 07:03:11 GMT

It’s really important; we have a real shot; there are a lot of ways we can fail.

AI for AI safety

Fri, 14 Mar 2025 05:27:24 GMT

We should try extremely hard to use AI labor to help address the alignment problem.

Paths and waystations in AI safety

Tue, 11 Mar 2025 18:36:29 GMT

On the structure of the path to safe superintelligence, and some possible milestones along the way.

When should we worry about AI power-seeking?

Wed, 19 Feb 2025 18:36:28 GMT

Examining the conditions for rogue AI behavior.

How do we solve the alignment problem?

Thu, 13 Feb 2025 22:35:05 GMT

Introduction to an essay series about paths to safe, useful superintelligence.

What is it to solve the alignment problem?

Thu, 13 Feb 2025 18:01:59 GMT

Also: to avoid it? Handle it? Solve it forever? Solve it completely?

Fake thinking and real thinking

Tue, 28 Jan 2025 19:51:40 GMT

When the line pulls at your hand.

Takes on “Alignment Faking in Large Language Models”

Wed, 18 Dec 2024 18:11:12 GMT

What can we learn from recent empirical demonstrations of scheming in frontier models?

Video and transcript of presentation on Otherness and control in the age of AGI

Tue, 08 Oct 2024 22:10:26 GMT

An attempt to distill down the whole “Otherness and control” series into a single talk.

(Part 2, AI takeover) Extended audio/transcript from my conversation with Dwarkesh Patel

Mon, 30 Sep 2024 18:56:30 GMT

Extra content includes: AI collusion; the nature of intelligence; concrete takeover scenarios; flawed training signals; tribalism and mistake theory; more on what good outcomes look like.

(Part 1, Otherness) Extended audio/transcript from my conversation with Dwarkesh Patel

Mon, 30 Sep 2024 18:23:15 GMT

Extra content includes: regretting alignment; predictable updating; dealing with potentially game-changing uncertainties; intersections between meditation and AI alignment; moral patienthood without consciousness; p(God).

Loving a world you don’t trust

Tue, 18 Jun 2024 16:29:25 GMT

Garden, campfire, healing water.

On attunement

Mon, 25 Mar 2024 12:08:45 GMT

Examining a certain kind of meaning-laden receptivity to the world.

Video and transcript of presentation on Scheming AIs

Fri, 22 Mar 2024 15:41:41 GMT

An intro to my work on scheming/”deceptive alignment.”

On green

Thu, 21 Mar 2024 17:08:57 GMT

Examining a philosophical vibe that I think contrasts in interesting ways with “deep atheism.”

On the abolition of man

Thu, 18 Jan 2024 13:51:12 GMT

What does it take to avoid tyranny towards the future?

Being nicer than Clippy

Tue, 16 Jan 2024 16:47:12 GMT

Let’s be the sort of species that aliens wouldn’t fear the way we fear paperclippers.

An even deeper atheism

Thu, 11 Jan 2024 17:07:37 GMT

Who isn’t a paperclipper?

Does AI risk “other” the AIs?

Tue, 09 Jan 2024 17:34:16 GMT

Examining Robin Hanson’s critique of the AI risk discourse.

When “yang” goes wrong

Mon, 08 Jan 2024 16:20:49 GMT

On the connection between deep atheism and seeking control.

Deep atheism and AI risk

Thu, 04 Jan 2024 18:49:59 GMT

On a certain kind of fundamental mistrust towards Nature.

Gentleness and the artificial Other

Tue, 02 Jan 2024 17:59:16 GMT

AIs as fellow creatures. And on getting eaten.

Otherness and control in the age of AGI

Tue, 02 Jan 2024 17:45:49 GMT

Introduction and summary for a series of essays about how agents with different values should relate to one another, and about the ethics of seeking and sharing power.

New report: “Scheming AIs: Will AIs fake alignment during training in order to get power?”

Wed, 15 Nov 2023 16:11:01 GMT

My report examining the probability of a behavior often called “deceptive alignment.”

Superforecasting the premises in “Is power-seeking AI an existential risk?”

Wed, 18 Oct 2023 19:16:07 GMT

Superforecasters weigh in on the argument for AI risk given in my report on the topic.

In memory of Louise Glück

Sun, 15 Oct 2023 02:31:57 GMT

“It was, she said, a great discovery, albeit my real life.”

Predictable updating about AI risk

Mon, 08 May 2023 21:35:36 GMT

How worried about AI risk will we be when we can see advanced machine intelligence up close? We should worry accordingly now.

Existential Risk from Power-Seeking AI (shorter version)

Wed, 22 Mar 2023 17:31:38 GMT

Building a second advanced species is playing with fire.

A Stranger Priority? Topics at the Outer Reaches of Effective Altruism

Tue, 21 Feb 2023 16:42:41 GMT

My dissertation.

Seeing more whole

Fri, 17 Feb 2023 03:53:28 GMT

On looking out of your own eyes.

Why should ethical anti-realists do ethics?

Thu, 16 Feb 2023 03:55:19 GMT

Who needs a system if you’re free?

On sincerity

Fri, 23 Dec 2022 08:40:30 GMT

Nearby is the country they call life.

Against meta-ethical hedonism

Thu, 01 Dec 2022 07:33:38 GMT

Can the epistemology of consciousness save moral realism and redeem experience machines? No.

Against the normative realist’s wager

Sun, 09 Oct 2022 08:34:50 GMT

If you find a button that gives you a hundred dollars if a certain controversial meta-ethical view is true, but you and your family get burned alive if that view is false, should you press the button? No.

Is Power-Seeking AI an Existential Risk?

Tue, 06 Sep 2022 10:58:53 GMT

Report for Open Philanthropy examining what I see as the core argument for concern about existential risk from misaligned artificial intelligence.

Video and Transcript of Presentation on Existential Risk from Power-Seeking AI

Sun, 21 Aug 2022 23:15:14 GMT

Video and transcript of a presentation I gave on existential risk from power-seeking AI, summarizing my report on the topic.

Dutch books, Cox, and Complete Class

Thu, 24 Mar 2022 00:00:00 GMT

Final essay in a four-part series on expected utility maximization (EUM). Examination of some theorems that aim to justify the subjective probability aspect of expected utility maximization (EUM), namely: Dutch Book theorems; Cox’s Theorem; and the Complete Class Theorem.

VNM, separability, and more

Mon, 21 Mar 2022 00:00:00 GMT

Third essay in a four-part series on expected utility maximization (EUM). Examines three axiomatic arguments for acting like an EUM-er: the von Neumann-Morgenstern theorem; an argument based on a very general connection between “separability” and “additivity”; and a related “direct” axiomatization of EUM in Peterson (2017).

Why it can be OK to predictably lose

Fri, 18 Mar 2022 00:00:00 GMT

This is the second essay in a four-part series on expected utility maximization (EUM). This part focuses on why it can make sense, in cases like “save one life for certain, or 1000 with 1% chance,” to choose the risky option, and hence to “predictably lose.” The answer is unsurprising: sometimes the upside is worth it. I offer three arguments to clarify this with respect to life-saving in particular. Ultimately, though, while EUM imposes consistency constraints on our choices, it does not provide guidance about what’s “worth it” or not — and in some cases, the world doesn’t either. Ultimately, you have to decide.

Skyscrapers and madmen

Wed, 16 Mar 2022 00:00:00 GMT

This is the first essay in a four part series on expected utility maximization (EUM). It presents a way of visualizing EUM that I call the “skyscraper model.” It also discusses (1) the failure of the “quick argument,” (2) reasons not to let your epistemic relationship to EUM stop at “apparently the math says blah,” and (3) whether being “representable” as an EUM-er is trivial (in the relevant sense, I don’t think so).

Simulation arguments

Fri, 18 Feb 2022 00:00:00 GMT

Examination of various arguments that we should assign significant probability to living in a computer simulation.

On infinite ethics

Sun, 30 Jan 2022 00:00:00 GMT

Infinities puncture the dream of a simple, bullet-biting utilitarianism. But they’re everyone’s problem.

The ignorance of normative realism bot

Mon, 17 Jan 2022 00:00:00 GMT

This essay lays out what I see as the strongest objection to normative realism: namely, that it leaves us without the right type of epistemic access to the normative facts it posits. To illustrate, I discuss various robots in unfortunate epistemic situations. Realism makes ours analogous.

Morality and constrained maximization, part 2

Mon, 10 Jan 2022 00:00:00 GMT

Second in a two-part series on whether morality falls out of instrumental rationality, if you do the game theory right. I discuss four objections to the morality in question: that it isn’t instrumentally rational; that it gives the wrong types of reasons for moral behavior; that it incentivizes threats and exploitation; and that it licenses arbitrarily bad behavior towards the sufficiently disempowered and unloved.

Morality and constrained maximization, part 1

Tue, 21 Dec 2021 00:00:00 GMT

First in a two-part series about whether morality falls out instrumental rationality, if you do the game-theory right. This part lays out of the basic structure of a prominent argument in favor.

Anthropics and the Universal Distribution

Sun, 28 Nov 2021 00:00:00 GMT

Essay on whether a certain sort of prior helps resolve tough questions in anthropics. My answer: maybe a few, but at high cost.

On the Universal Distribution

Fri, 29 Oct 2021 00:00:00 GMT

Essay examining a certain sort of fundamental prior, called the “Universal Distribution.”

In defense of the presumptuous philosopher

Thu, 30 Sep 2021 00:00:00 GMT

Final essay in a four-part series, explaining why I think that one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part discusses some prominent objections to SIA.

An aside on betting in anthropics

Thu, 30 Sep 2021 00:00:00 GMT

This essay is the third in a four-part series, explaining why I think that one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part briefly discusses betting in anthropics. In particular: why it’s so gnarly, why I’m not focusing on it, and why I don’t think it’s the only desiderata.

Telekinesis, reference classes, and other scandals

Thu, 30 Sep 2021 00:00:00 GMT

This essay is the second in a four-part series, explaining why I think that one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part focuses on objections to SSA.

Learning from the fact that you exist

Thu, 30 Sep 2021 00:00:00 GMT

This is the first essay in a four-part series explaining why I think one prominent approach to anthropic reasoning (the “Self-Indication Assumption” or “SIA”) is better than another (the “Self-Sampling Assumption” or “SSA”). This part lays out the basics of the debate.

Can you control the past?

Fri, 27 Aug 2021 00:00:00 GMT

I think that you can “control” events you have no causal interaction with, including events in the past, and that this is a wild and disorienting fact, with uncertain but possibly significant implications. This essay attempts to impart such disorientation.

In search of benevolence (or: what should you get Clippy for Christmas?)

Mon, 19 Jul 2021 00:00:00 GMT

What is altruism towards a paperclipper? Can you paint with all the colors of the wind at once?

On the limits of idealized values

Mon, 21 Jun 2021 00:00:00 GMT

Contra some meta-ethical views, you can’t forever aim to approximate the self you would become in idealized conditions. You have to actively create yourself, often in the here and now.

Problems of evil

Mon, 19 Apr 2021 00:00:00 GMT

Is everything holy? Can reality, in itself, be worthy of reverence?

The innocent gene

Sun, 04 Apr 2021 00:00:00 GMT

Richard Dawkins imagines genes as selfish agents. I find it evocative (and in some cases, tragic) to stay within his gene-centered framing, but to remember that the genes are passive patterns, that don’t want to survive, that know not what they do — even as they structure the violence and beauty swirling around them.

The importance of how you weigh it

Sun, 28 Mar 2021 00:00:00 GMT

Much of normative ethics centers on which considerations matter, and why. But often, it makes a bigger difference *how much* a consideration matters. I worry that neglecting this leads to the wrong sorts of arguments and points of focus.

On future people, looking back at 21st century longtermism

Mon, 22 Mar 2021 00:00:00 GMT

An intuition pump for a certain kind of “holy sh**” reaction to existential risk, and to the possible size and quality of the future at stake.

Against neutrality about creating happy lives

Sun, 14 Mar 2021 00:00:00 GMT

Making happy people is good. Just ask the golden rule.

Care and demandingness

Sun, 07 Mar 2021 00:00:00 GMT

We wouldn’t expect prudence to be “not demanding.” Why would morality be different?

Subjectivism and moral authority

Mon, 01 Mar 2021 00:00:00 GMT

Reflections on the sense in which subjectivism about meta-ethics deprives morality of authority.

Two types of deference

Sun, 21 Feb 2021 00:00:00 GMT

Two different ways of thinking about the epistemic position of future people: better informed about questions we ask using today’s concepts, vs. incomprehensible.

Contact with reality

Sun, 14 Feb 2021 00:00:00 GMT

If God were playing a concert, would you take some hit to your overall pleasure in order to go? Against experience machines.

Killing the ants

Sun, 07 Feb 2021 00:00:00 GMT

If you kill something, look it in the eyes as you do.

Believing in things you cannot see

Sun, 31 Jan 2021 00:00:00 GMT

Lots of ethical life, especially in a big world, rests on the ability to treat things you can’t see as real.

On clinging

Sun, 24 Jan 2021 00:00:00 GMT

How can “non-attachment” be compatible with care? We need to distinguish between caring and clinging.

Actually possible: thoughts on Utopia

Mon, 18 Jan 2021 00:00:00 GMT

There are oceans we have barely dipped a toe into. There are drums and symphonies we can barely hear. There are suns whose heat we can barely feel on our skin.

Shouldn’t it matter to the victim?

Sun, 10 Jan 2021 00:00:00 GMT

Shouldn’t deontological distinctions between types of harm reflect something that matters *to* the potential victims of that harm? But they don’t.

The despair of normative realism bot

Sun, 03 Jan 2021 00:00:00 GMT

We don’t need non-natural normative facts to tell us what to do. We can decide for ourselves.

A ghost

Sat, 26 Dec 2020 00:00:00 GMT

On a thought experiment I find useful in thinking about what to do.

Alienation and meta-ethics (or: is it possible you should maximize helium?)

Sun, 20 Dec 2020 00:00:00 GMT

Can what you should do hold no appeal whatsoever?

Wholehearted choices and “morality as taxes”

Sat, 12 Dec 2020 00:00:00 GMT

Reimagining Peter Singer’s drowning child from the perspective of care rather than guilt.

Thoughts on being mortal

Sun, 06 Dec 2020 00:00:00 GMT

You can’t keep any of it. The only thing to do is to give it away on purpose.

Grokking illusionism

Sun, 29 Nov 2020 00:00:00 GMT

Trying to see through the eyes of people who think consciousness is an illusion.

The impact merge

Sun, 22 Nov 2020 00:00:00 GMT

Against hitching together your desire to accomplish things and your desire to do good.

Thoughts on personal identity

Mon, 16 Nov 2020 00:00:00 GMT

On living in a glass tunnel, and on open air.

How core is confusion about consciousness?

Sun, 08 Nov 2020 00:00:00 GMT

If we’re confused about consciousness, what else are we confused about?

To light a candle

Sun, 01 Nov 2020 00:00:00 GMT

In my last post, I wrote about an ethos expressed in A Wizard of Earthsea — an ethos that I find deeply resonant, but that I also think worth some scrutiny. This post attempts some of that scrutiny. Here is an effort to summarize some part of this ethos: turn outwards to the world — […]

The gestures of trees

Sat, 24 Oct 2020 00:00:00 GMT

Recently I read Ursula K. Le Guin’s A Wizard of Earthsea. The book has a certain kind of ethos, which I found moving and beautiful, but which can seem — to some, and to some parts of me — questionable. In this post, I’m going to try to gesture at that ethos; in the next, I […]

Mistaking the plot

Sat, 17 Oct 2020 00:00:00 GMT

I started reading The Guns of August, Barbara Tuchman’s account of the beginning of World War I. One passage in particular struck me. She describes the German general Alfred von Schlieffen, who authored, in 1905 and 1906, the plan that Germany eventually employed in attacking France in 1914, via Belgium on the right wing. Image […]

How much computational power does it take to match the human brain?

Tue, 15 Sep 2020 08:42:00 GMT

Report for Open Philanthropy on the computational power sufficient to match the human brain’s task-performance. I examine four different methods of generating estimates in this respect.