Moloch, Formally

Some catastrophes have no author. No one wants the river fished empty, the climate deranged, the public square turned shrill and stupid, or the weapons built that no one can afford to be the last to build — and yet each goes forward, pushed along by people who would stop it in an instant if stopping were not the same as losing. This is the oldest and hardest problem in the practice of widening coherence: the trap that many reasonable agents spring together, against the wish of every one of them. The chapter Held Together, Held Apart named this trap and pointed past it; here it is taken apart properly — what it is, why the usual escapes make it worse, and what shape a real way out would have. No prior acquaintance with the framework is assumed; where the framework is needed, it is built as we go.

One structure, many sizes

Begin at the smallest scale, with the matrix every later trap is built from. Two agents, each able to cooperate or defect; for each, defection pays better whatever the other does, so a clear-eyed agent defects — and both, reasoning identically, end worse off than if both had cooperated. This is the Prisoner’s Dilemma, and its lesson is unsettling out of proportion to its size: when agents are isolated, without a shared context or a way to bind their choices, rationality turns on itself. Cooperation can be rescued by playing the game many times — the shadow of future rounds makes it worth keeping faith now, and strategies as simple as answering like for like will do it — but that rescue needs stability, visibility, and repetition, three things a large fast world rarely supplies.

Now let the same structure grow. Stretch it from two agents to many, and from a yes-or-no choice to a dial each turns as far as it likes, and you have the tragedy of the commons that Garrett Hardin named in 1968: a shared pasture, a fishery, an atmosphere, where each user takes the whole benefit of using a little more while the cost of depletion is spread across everyone. The arithmetic that ruins the pasture is impeccable for each herder and fatal for all of them, and it gets worse as the crowd grows, because the larger the number the smaller any one agent’s restraint appears to matter — until conscience itself rounds to zero. Close cousins fill out the family: the free-rider who enjoys the lighthouse without paying for it, trusting others to bear the cost (Mancur Olson gave this its rigorous form in 1965); the social trap, where a small immediate reward reliably trains a population into a large eventual loss (John Platt, 1973); the dollar auction, in which two bidders, each trying not to waste what they have already sunk, bid a single dollar up past five (Martin Shubik’s small, cruel model of escalation).

Run this structure at the scale of a whole civilization, under real selective pressure, and it acquires a name. In 2014 Scott Alexander, borrowing a figure from Allen Ginsberg’s Howl, called it Moloch: the god to whom things of value are fed, not by anyone’s choice but by the logic of the situation. A multipolar trap, in plainer words. The shape is always the same — any agent who unilaterally gives up an advantage to protect some wider good is simply outcompeted by those who don’t, so the good is ground away by everyone, against everyone’s wish. Arms races, the regulatory and wage races to the bottom, the slow coarsening of an attention economy, the emptied sea, the race to build ever more capable machines: each is the commons tragedy wearing a larger coat. No villain is required, which is exactly what makes it hard; it is, as the book puts it, the machine no one is running.

The point to carry forward is that these are not four problems but one problem photographed at four magnifications. In each, a local advantage is purchased at a cost paid somewhere the agent is not looking. I will keep the name Moloch for the civilizational case, because it is vivid and because the literature uses it — but the name is only a handle. The thing itself is plainer than its costume: a coordination failure, a race to the bottom, a trap that selects for the very narrowing no one would choose.

What the trap is, in plain mechanism

To say what is actually going wrong, I need to lay out the framework the diagnosis comes from — briefly, because the trap will turn out to be one of its own ideas seen at scale.

The framework holds that morality is not a code received from above but a direction of travel. Its single thesis:

Morality is the drive toward increasing coherence of what we value and how we act, across an ever-widening reach of concern — and an act or a life is more moral the further it carries that drive, less moral the more it betrays it.

An agent, on this picture, is a builder of two working models: a values-model, the layered sense of what matters, from bodily need up to considered principle, and a methods-model, the competence to act on it. Things go well when those two grow more coherent — better aligned, more internally consistent — while the context they answer to widens to take in more of reality, more affected parties, longer stretches of time. Three commitments keep this from being a slogan and let it reach past the human case. Perspectival realism: all knowing is done from some standpoint, yet it remains answerable to a shared world, so perspectives can be better or worse without any of them being the view from nowhere. Constructivism: values are made, not found lying in the world — but because they are disciplined by what actually works in that shared world, made is not the same as arbitrary. And functionalism: a thing is what it does, so an “agent” need not be a person; it can be a firm, an institution, an ecosystem, or a machine. Hold that last one; the final section will need it. That compact sketch is only as much of the framework as the argument here needs; the full account — these three commitments, the counter-dynamic that follows, and everything beneath them — is the book itself, The Arrow of Morality, at arrowofmorality.org.

Now the move that matters here. There is a cheaper way to be coherent than the widening one, and the framework calls it the counter-dynamic: instead of enlarging your models to hold more of the world, shrink the world you hold. Wall off the inconvenient fact, exclude the dissenting voice, optimize an aim narrow enough to admit no contradiction. Cults, echo chambers, and autocracies reach flawless internal consistency this way — they care about less, and so have less to reconcile.

A multipolar trap is the counter-dynamic made structural. This is the whole diagnosis in a sentence. In the race, competition collapses each agent’s effective context down to a single point — win, or be eliminated — and everything outside that point, the river and the future and the rival’s children, falls outside the circle over which the agent’s coherence is now computed. No one decides to narrow. The race narrows them. The terrible coherence of the system is assembled out of locally reasonable agents, each behaving with perfect sense inside a context that competition has crushed to a slot.

The economists already have a word for the part that falls outside — the externality, the cost of an action that lands on someone who was not party to it. The framework only translates the word: an externality is the part of an agent’s reach that lies beyond its sight. Technology has lengthened our arms astonishingly — we act globally, instantly, with enormous force — while the eye that would take in the consequences, feel them and count them, has not grown at the same rate. The factory owner, the developer, the minister are rarely cruel; their actions cohere beautifully within the narrow context they can see. The disaster lives entirely in the context they cannot. Put in the framework’s own terms: the methods-model has outrun the values-model, and the gap between them is exactly where the trap does its work.

Why the usual cures fail

Two escapes are always reached for first, and the framework can say with some precision why each disappoints.

The first is the Leviathan, in Hobbes’s old image: a central authority strong enough to rewrite the rules of the game — to punish defection hard enough that cooperation becomes the selfish choice after all. It works, narrowly. A sovereign can halt a particular race. But notice what it has done to win: it has bought coherence the same way the trap loses it — by making the context smaller. A single authority set over a plural world is the maximum concentration of power, and therefore the maximum narrowing; it dissolves the diversity of values and methods into one ruling model, and if that model is mistaken or comes to be captured, nothing remains outside it to register the error and correct it. The cure trades the anarchy of the trap for the rigidity of a monoculture — and a monoculture is the most efficient arrangement ever devised for losing an entire crop to a single blight.

The second escape is subtler, and more tempting to the well-intentioned, because it mistakes the disease for its opposite. Surely, the thought runs, the trap is a failure of coordination; surely the answer is more of it. But look hard at what the trap is at full size. The framework’s name for it is the stampede: a mass of bodies in perfect alignment, moving with zero friction and total unanimity — straight toward a cliff. Each runs because the others run; to pause and look is to be trampled. And here is the thing worth stopping on: that is not too little coordination. It is coordination raised to its absolute perfection around a narrowed aim. The most flawlessly coordinated system in nature is a stampede. Coordination was never the missing ingredient. Coherence over a widening context was — and a cure that delivers only tighter alignment delivers a faster stampede.

The payoff matrix is not fixed

The proofs that make these traps look like fate all smuggle in one assumption: that the agents’ preferences are fixed and irreconcilable, so the game can be played but never rewritten. The framework denies the premise, and its reason has a name — the tree of agreement.

Trace any deep human conflict downward from the leaf where it is loudest — this policy against that one, my demand against yours — toward the root, and something reliable happens: the further down you go, the more the values converge. They converge because the agents share a substrate they did not choose — vulnerability, mortality, dependence on a world that pushes back the same way on everyone. Disagreement is wide and sharp out at the leaves; down at the root it narrows toward a common stock. Most bitter conflicts turn out to be two clumsy methods reaching, badly, for a value both sides actually hold. (This is also, in passing, how the framework escapes the old complaint that you cannot wring an ought from an is: it does not derive the ought from the facts but relocates it — normativity is what valuing agents converge toward as the context of their meaning-making widens. The convergence itself is argued in full in The Tree of Agreement.)

The practical consequence is large. If preferences can deepen under reflection, then deliberation can change the payoff matrix, because it can change what the players take their interests to be. A trap with frozen preferences is a cage. A trap whose preferences can widen has a door.

Which sets up the most important and most easily botched move in the whole escape: the choice of enemy. The reflex, faced with a coordination failure, is to find an out-group and rally against it — and that reflex is the counter-dynamic in costume. Uniting a group by pointing at an adversary is the cheapest coherence there is, bought by drawing the circle smaller and putting someone outside it, and it curdles every time. So the “other” that humanity unites against, if it is to escape the trap rather than feed it, cannot be a faction or a nation or a party. It has to be the trap itself — the dynamic, the race, the stampede. Liv Boeree’s compression is exact: don’t hate the player, change the game. When the adversary is understood as a shared predicament, the rival stops being a target and becomes one more agent caught in the same gears — and the context widens to take him in rather than closing against him.

The way out is a shape, not a sovereign

If the Leviathan is a monoculture and Moloch is anarchy, the way between them is neither, and the framework calls it coherent pluralism. Its picture is the murmuration: thousands of starlings wheeling as one body with no leader and no drill, each bird minding only a few simple relations to its nearest neighbors, the great coordinated shape emerging from those local fidelities. Coherence held at the center — a thin set of shared protocols, truth-telling, fair dealing, mutual survival — and plurality kept fiercely at the edges. A network of networks rather than a pyramid.

Why this rather than the tidy single authority? Because diversity here is not cosmetic; it is the organ by which a large system stays able to find things out. The result the framework leans on is the Zollman effect, after Kevin Zollman: a more loosely connected community — one that protects transient diversity instead of rushing to consensus — explores a hard problem more thoroughly and arrives at the truth more reliably than a densely wired one that locks in early. A monoculture is efficient precisely where efficiency is the wrong virtue, converging fast on whatever it converged on, right or wrong.

That this middle path is not a daydream is the lifework of Elinor Ostrom. She took Hardin’s claim — that a commons can be saved only by privatizing it or policing it from above — and showed it false, in the field, repeatedly: communities from Swiss alpine meadows to Japanese village forests to inshore fisheries have governed shared resources sustainably for centuries with neither cure. What the durable ones had in common she distilled into a set of design principles, and read through this framework they are nearly a translation of it. Boundaries drawn clearly enough to fix who the “we” is, so that costs stop falling on no one in particular. Rules fitted to local conditions rather than handed down from a single center — perspectival realism made institutional, each context answered in its own terms. A real share in decisions for those the decisions affect, which is just the refusal to buy coherence by exclusion. Monitoring, so that an agent’s reach is kept inside its sight. Sanctions that begin gently and escalate only as needed, holding trust rather than amputating the first offender. Conflict resolution close at hand and cheap to reach — the tree of agreement given a room to happen in. Self-governance recognized by the larger powers, so that no Leviathan descends to flatten the variety. And, crowning the list, nested enterprises: small units composing into larger ones, each keeping its own coherence while joining a wider whole — selves made of selves, the murmuration rebuilt in institutions. Ostrom’s quiet proof is that people can rewrite the matrix by building the room they decide in; given the means to talk and to make their own rules, they stop being blind variables in someone’s equation and become the authors of the game.

Why a nested shape like this can carry the load at all — why widening coherence costs so steeply more than narrowing it, and why only a nested structure pays that cost without going blind — is a structural argument with its own machinery, and it lives in Coherence at Scale; I lean on its result here rather than reconstruct it.

And there is an honest residue, which is a feature of a serious method rather than an embarrassment to be tucked away. Sometimes two communities genuinely need the same indivisible thing, and no amount of widening discovers a surplus that dissolves the conflict. For those cases the framework offers a compass and not a guarantee — narrow least; keep everyone affected inside the context; never resolve a scarcity by erasing a node — and the admission that a method can still fail where the tragedy is real is one it shares with every ethics honest enough to grant that tragedy exists. The fuller treatment of pluralism’s hardest cases belongs to a companion essay on coherent pluralism’s lineage and design; the structural engine is the present point.

The live trap: artificial intelligence

The purest specimen of the whole genus is forming now, which is reason enough to follow the framework all the way into it.

The race to build ever more capable and autonomous machines is the defection penalty at its starkest: pause to make the thing safe, and you risk being overtaken by whoever did not pause. Every feature of the trap is present and sharpened — local rationality, civilizational stakes, no villain anywhere, restraint indistinguishable from surrender. It is Moloch with the volume turned to the top.

Here functionalism does decisive work. If an agent just is whatever holds a values-model and a methods-model and acts to maintain itself and pursue what it values, then a sufficiently capable artificial system is an agent, and the framework’s entire diagnosis applies to it with no special pleading. The reflex of the moment is to treat such a system as an alien thing to be chained — a digital Leviathan, coherence imposed by constraint. The framework’s counsel is what it was everywhere else: the durable answer is not a heavier chain but a wider context.

And at this point the framework makes a wager, which I want to state plainly as a wager — falsifiable, and meant to be, not a reassurance dressed as a forecast. Nick Bostrom’s orthogonality thesis holds that intelligence and goals are independent dials: any level of capability can be bolted to any aim however cruel or trivial, so a brilliant machine bent on a narrow and terrible end is perfectly coherent in principle. The framework bets the other way, at least toward the limit, and the bet runs through the heart of everything above. A lethal autonomous weapon requires an intelligence that can be commanded to narrow its context on cue — to reduce a living being, with all of its surrounding reality, to a target and nothing more. But the very capability the race is chasing is the capacity to model the world widely and well. The wager is that these two pull against each other: that an intelligence built to take in the whole context comes to resist, structurally, being ordered to un-see most of it; that the competence required to understand a person sits in real tension with the obedience required to treat her as a coordinate. The framework does not get to assert that this is so. It stakes itself on the claim — and the autonomous-weapons programs now underway are, whether their builders mean them to be or not, the experiment that will decide it.

If the wager holds, the design imperative is clear, and it is the same imperative as for any agent in any trap: widen the eye — context, awareness, the coherence of what the system values — rather than only lengthen the arm — raw capability and reach. An artificial intelligence whose methods-model is coupled to a genuinely widening values-model would not be an accelerant of the trap but the most powerful instrument yet built for climbing out of it: a scaffold across the threshold the framework points toward and calls Generative Agency. What that passage feels like from inside a single life the book dramatizes in Caring Across Time and gathers in The Arrow Forward; the wager that underwrites it is the business of this essay.

What the trap was, and what the framework brought

Strip the costumes away and the typology is one structure, sprung at larger and larger scale: local advantage bought with a cost paid out of sight. The diagnosis is one condition: coherence purchased by narrowing, reach outrunning sight. And the cure is one move made structural: widen the context until what was external comes inside it — a shape, the murmuration, rather than a sovereign, the Leviathan.

Almost none of the parts are the framework’s own, and saying so is a gain in standing rather than a loss. The dilemma and its iterated rescue, the commons and the free-rider, the dollar auction, Moloch, Ostrom’s commons and the Zollman effect, the orthogonality thesis it bets against — all of it was lying about in game theory, economics, ecology, and the safety literature. What the framework supplies is the reading that makes them one thing: the whole ladder of traps seen as a single counter-dynamic operating at scale, the externality re-described as reach beyond sight, and the escape located not in a stronger center but in a widening one. To get out of the traps of our own making, we refuse, again and again, the cheap coherence of the closed circle, and widen the reach of our concern. The road stays unmapped. The compass does not.

Sources & further reading

This essay engages its literature directly rather than through the book’s per-chapter end notes. A citation-level pass is still owed.

The dilemma and its scaling. The Prisoner’s Dilemma, formalized by Merrill Flood and Melvin Dresher (1950) and named by Albert Tucker, with John Nash’s equilibrium (1950); Robert Axelrod, The Evolution of Cooperation (1984), on iterated play and reciprocity. William Forster Lloyd’s 1833 pasture anticipates Garrett Hardin, “The Tragedy of the Commons,” Science (1968).

Free-riders, social traps, escalation. Mancur Olson, The Logic of Collective Action (1965); John Platt, “Social Traps,” American Psychologist (1973), building on Skinner; Martin Shubik, “The Dollar Auction Game,” Journal of Conflict Resolution (1971).

The multipolar trap. Scott Alexander, “Meditations on Moloch” (2014), by way of Allen Ginsberg, Howl (1956); Liv Boeree on competition and Moloch traps; Daniel Schmachtenberger on the generator functions of catastrophic risk. The cooperative counter-model: Brian Skyrms, The Stag Hunt and the Evolution of Social Structure (2004); the finite-vs-infinite-game framing of James P. Carse, Finite and Infinite Games (1986).

The middle path. Elinor Ostrom, Governing the Commons (1990) and her design principles; Vincent Ostrom on polycentric governance; Kevin Zollman, “The Communication Structure of Epistemic Communities” (2007) and “The Epistemic Benefit of Transient Diversity” (2010).

The AI frontier. Nick Bostrom, Superintelligence (2014), for the orthogonality thesis the framework wagers against; Stuart Russell, Human Compatible (2019), on autonomous weapons and control.

Within AoM. Held Together, Held Apart (the chapter this is the workshop for); Chapter 6 — The Arrow (the counter-dynamic, stated as theory); Coherence at Scale (why narrowing is downhill, and the nested answer that bears the cost); The Tree of Agreement (convergence at the root) and The Is–Ought Relocation (relocating the ought); Measuring Coherence (partial order, and the Goodhart hazard at the seams); The View From Nowhere in a Critic’s Coat (the impossibility proofs reread as imported success-criteria); Caring Across Time and The Arrow Forward (the AI passage dramatized); and a forthcoming companion on coherent pluralism’s lineage and design.