A sunless summer, two founding monsters
In the summer of 1816, in a villa on the shore of Lake Geneva where a sunless season (the one that followed the eruption of Mount Tambora, the so-called “year without a summer”) kept the guests indoors, Lord Byron challenged his companions each to write a ghost story. Mary Shelley, eighteen years old, produced the only tale to outlive the game: Frankenstein; or, The Modern Prometheus, published anonymously in 1818. […] Two centuries after Lake Geneva, the polarity reverses term for term: we in our turn build a being made of language, eloquent and faceless and far from fleeing it as Victor fled his creature, we hasten to welcome it in the place of our neighbor.
Two details tighten the knot further. Mary Shelley was the daughter of the philosopher William Godwin and of Mary Wollstonecraft, one of the first philosophers to argue for women's rights, who died ten days after giving birth to her: a novel haunted by creation, parentage and abandonment carries, beneath it, the shadow of that birth. And Frankenstein was not the only monster born of that rainy week: John Polidori, Byron's personal physician, also of the party, wrote to the same challenge The Vampyre (1819), the first tale to fix the figure of the seductive, romantic vampire, nearly eighty years before Dracula. Two founding monsters of Western popular culture were born of the same stormy evening. One last gap is worth noting: popular culture did not keep the novel's creature, eloquent, self-taught, a reader of Milton, but the mute, grunting one of James Whale's 1931 film with Boris Karloff, the exact reverse of what Mary Shelley wrote and one more instance of something this book observes elsewhere: legend rarely keeps what the text actually says.
Olimpia, the automaton that heard nothing
In Hoffmann's The Sandman, Nathanael falls madly in love with Olimpia, who listens to him for hours without ever interrupting, sighing at just the right moment; she turns out to be an automaton. The tale carries the exact diagnosis: Olimpia did almost nothing and it is that very emptiness that let Nathanael deposit in her the intelligence, the attentiveness and the love he wanted to find there.
This 1816 tale had an afterlife the book does not follow. In 1919, Freud made it the centerpiece of his most famous essay, The Uncanny (Das Unheimliche): it is precisely from Olimpia that he theorizes the unease proper to what resembles the human without quite being it, a century before the English term uncanny valley named the same sensation in front of a synthetic face. Freud is cited elsewhere in this book for his “narcissistic wounds”; it is the same Freud, on the same tale, who closes the loop here. The scene also had a second life, sung: Jacques Offenbach made it the central act of The Tales of Hoffmann (1881), with the doll's aria “Les oiseaux dans la charmille” (the Doll Song), one of the most treacherous coloratura pieces in the repertoire, where the soprano must imitate, in her own voice, the jolts of a mechanism running down and sometimes needing to be wound again mid-scene, an automaton singing its own automatism, still applauded today.
Jeeves and its digital grandchild, Ask Jeeves
Literature gave this debt its most amiable face: Jeeves, Wodehouse's perfect valet, to whom his master Bertie Wooster never decided to delegate everything. The delegation happened by small touches […] no tiny permission was ever revoked.
The character had a direct, literal descendant: in 1996, David Warthen and Garrett Gruener launched a natural-language search engine they named Ask Jeeves, top hat and all, promising to answer any question the way the perfect valet answered any need. The site renamed its engine Ask.com in 2006, once the promise of an omniscient butler was overtaken by the far more mundane reality of keyword search. One amusing note Wodehouse never quite explains: the comic engine of the entire series rests on Bertie never fully understanding how his own household runs, a century-early anticipation of this book's own point about delegated competence becoming opaque to the very person who delegated it.
Two presidencies, two shadow astrologers
Between 1989 and 1995, French President François Mitterrand regularly consulted the astrologer Elizabeth Teissier before certain decisions, going as far as asking her, after the Gulf War, which day would be best to “intervene.” Delegating a real decision to an opaque oracular device did not wait for the conversational agent; the agent only generalizes access to it.
The case is far from isolated and the other great nuclear democracy of the Cold War offers an even better-documented parallel. After the 1981 assassination attempt, Nancy Reagan began consulting the astrologer Joan Quigley, who stayed in near-daily contact with her for seven years. The affair broke only in 1988, in Chief of Staff Donald Regan's memoir: by his account, Air Force One's departure and landing times, debate dates and even the scheduling of summits with Gorbachev were set according to the stars. A New York tabloid ran the headline “Astrologer Runs The White House.” Both cases share the same structure this book describes elsewhere: opacity is not a flaw in the oracle, it is the appeal, a box that cannot be interrogated is exactly what lets whoever consults it stop carrying the weight of the decision alone.
EURISKO: from gaming a war game to Cyc
In 1981, EURISKO, Douglas Lenat's program that invented and scored its own heuristics, won the US championship of the naval-strategy game Traveller by fielding a fleet so aberrant that the organizers changed the rules; it won again in 1982, having discovered among other things that it could sink its own damaged ships and was excluded from the competition afterwards.
What Lenat drew from this is worth following to the end. Rather than further refining the art of gaming a game's rules, he spent the following four decades on the opposite wager: Cyc, begun in 1984 and spun off in 1994 as the company Cycorp, a common-sense knowledge base hand-encoded, statement by statement, to finally give machines what EURISKO never had: a grip on what its own outputs meant in the world, rather than only on how to win by a scoring rule. Lenat pursued that same project until his death in 2023, one man, one diagnosis, held for forty years, from “gaming the rules of a war game” to “trying to give a machine common sense.”
The Butlerian Jihad, from Samuel Butler to Dune
The maximal alternative, outright refusal, has its literary emblem: in Dune, a past revolt against thinking machines, the “Butlerian Jihad,” proscribes a whole class of artifacts and its law commands “Thou shalt not make a machine in the likeness of a human mind.” Herbert did not invent the name: it salutes Samuel Butler, whose satire Erewhon (1872) already argued, in a naturalist's voice, that machines evolve like species and will end up dominating man; the imaginary country concludes by banning any machine beyond a threshold set centuries earlier. That is the ban of an entire category, not the grading of its use; we take the other road.
The gesture of naming rewards tracing to its source. Butler was not improvising: as early as 1863, in a letter to the New Zealand newspaper The Press titled “Darwin among the Machines,” he applied to machines the very grammar of On the Origin of Species, published four years earlier. Erewhon turned this, in 1872, into a three-chapter treatise, the “Book of the Machines,” arguing in the exact tone of a naturalist rather than a pamphleteer that machines vary, reproduce through man and will one day be selected by an evolution owing nothing further to their maker. The imaginary country draws the most radical conclusion imaginable: ban any machine newer than a threshold fixed centuries earlier, a step even today's most cautious advocates of AI governance do not propose. Herbert borrows the bone of the argument, not its solution: in his hands, the ban becomes an armed religion, held to for a century, its common law compressed into a single sentence. The name has since spilled out of fiction: in 2021, essayist Erik Hoel published “We need a Butlerian Jihad against AI,” one of the first notable uses of the term outside Dune's universe to call, without irony, for a moratorium on AI research. The literary emblem of total refusal keeps circulating, long after the novel that coined it, whenever the option of outright prohibition returns to the table.
Yves Robert, the parable and a title with a double bottom
The French pun says it already: un éléphant, ça trompe énormément, literally “an elephant, that deceives enormously,” playing on trompe, which names both the trunk and the verb to deceive. The animal chosen to escape anthropomorphism thus finds itself, by its very name, bound to deceive as well, fitting for something one only ever grasps in part: the joke excuses nothing, no figure, human or animal, replaces the observation of acts.
The wordplay is craftier than it looks, landing on its feet twice over. Yves Robert's 1976 film (released in English as Pardon Mon Affaire) is not about elephants at all: it is a comedy about four men in their forties tangled in affairs each of them misreads, a romantic blindness the title compresses into an idiom already old before the film existed, where trompe slides from the organ to the verb. The film lends this book its title without lending it its plot: what travels is the shape of the pun, not the story. And that shape lands squarely on the parable the book chooses to picture itself: blind men each feeling one part of an elephant and mistaking their partial grip for the whole animal. The elegance of the choice lies in a doubling rarely noticed: the very animal enlisted to compose partial grips into a whole carries, in its French name, the trace of the same flaw it is meant to cure. No figure escapes entirely what it describes; the book knows this and says so through the joke rather than staying silent about it.
The Machine Stops, predicted in 1909, reread in 2020
Forster had given, as early as 1909, its extreme version in The Machine Stops: a whole humanity has entrusted its every need to a world Machine it no longer understands, until the day the Machine stops and no one knows how to live without it. Our answer is not abstinence but continuity: a local recourse one can open and repair oneself.
The story rewards reading in full: its accuracy outstrips what a reader today half-remembers of it. Published in November 1909, six years before radio broadcasting existed, The Machine Stops imagines a humanity living in isolated underground cells, each fitted with a device that is part videophone, part instant messenger: press a button, see and hear a correspondent through a “blue plate,” deliver a lecture heard by thousands without leaving your chair. Vashti, the story's central figure, almost never leaves her cell and finds direct contact with another body nearly obscene; all her knowledge arrives secondhand, relayed by the Machine, never checked against the thing itself. The story's real engine is not dependence as such but its precise mechanics: the Machine does not stop all at once, it degrades in increments nobody wants to name, until total failure hits a civilization that had forgotten, generations earlier, how to repair it. That is exactly the mechanism this book describes around proprietary software: owning access is worthless if no one still knows how, or is allowed, to open the hood. The story saw a revival of readers in the spring of 2020, when critics confined to their homes, talking to each other through screens, recognized their own situation in the one Forster had imagined a hundred and eleven years earlier.
A 1:1 map, signed by an imaginary forger
The Argentine writer Jorge Luis Borges took the idea to its limit in a one-paragraph parable: an empire draws a map at the exact scale of the territory, so faithful it covers the whole of it, before being abandoned to the deserts, grown useless. A model is worth only what it leaves out; a representation that omits nothing no longer represents, it duplicates. The common trap is the mirror image: mistaking the map for the territory. The British statistician George Box gave this intuition its working formula: “all models are wrong, but some are useful.” We dispute the very word: a model is never wrong, nor is it right; it is built and the category of true and false applies to it no more than to a screwdriver. The only question that matters afterward is whether it serves to predict, to act, to decide. It is this shift, from true to useful, that will later license judging an agent by its acts rather than by an intelligence no one can establish.
The fable has its own history as a forger, one that doubles exactly what it is about. Borges does not sign “On Exactitude in Science” in his own name: he attributes it to a certain Suárez Miranda, the invented author of an equally invented travel treatise, Viajes de varones prudentes, dated 1658, Lérida. The parable is thus itself a false map of a source that never existed, exact in its bibliographic form (author, title, place, year) and fictional in its content: it performs, in a single line of attribution, the very trick it describes as its subject. Its afterlife took a still stranger turn. In 1981, the French philosopher Jean Baudrillard opens Simulacra and Simulation by quoting this same text, but reverses its meaning: in Borges, it is the map, worn down by the centuries, that rots in the desert while the territory survives it; in Baudrillard, it is the other way around, the territory whose shreds slowly rot across the map, the map now preceding the territory it is supposed to represent. The misreading is likely deliberate: it serves exactly the thesis of “hyperreality” Baudrillard develops, where representation stops following the real and starts outrunning it. What remains, worth noting in a book that takes map and territory as its central image, is a vertigo of its own: the founding text of the genre was itself misquoted by its most famous reader and neither version has ever needed correcting for both to keep illuminating something true.
Harness: from the weaving loom to the agent engine
The word makes a round trip worth telling. Harnais appears in French as early as 1155 (“herneis,” from Old Norse hernest, an army's provisions), then denotes in turn a warrior's armor (1160), clothing (1228), a horse's tack (1230), and, from 1765, the full assembly of parts in a weaving loom: the French word already carried, long before computing, a sense of mechanical assembly around a working core. English harness descends from it, borrowed around 1300 in that same broad sense before narrowing, within a few decades, to draft-horse tack. Agentic AI's vocabulary borrows from English rather than French, though: the translation attempt “harnais d'agent” surfaced only in 2026, its coinage itself disputed among practitioners, a sign of a still unsettled term. We therefore say engine (or, in the prologue, arrangement), without claiming to translate it.
The history rewards one more step than the book's own note allows. “Herneis” descends, via Old Norse hernest, from a compound literally naming an army's provisions and equipment on campaign, which explains its quick slide toward armor and then horse tack: in all three senses, the word is about equipping a body, human or animal, for action. The move to the weaving loom in 1765 is not an arbitrary extension but a continuation of the same technical sense: in weavers' own vocabulary, a loom's “harness” is literally the set of cords and heddles that raise the warp threads in a programmed order, the earliest documented sense of a mechanical device executing, with no hand guiding each single move, a sequence of instructions coded in advance. The word thus already carried, three centuries before the computer, the exact architecture it names today: a control core surrounded by the parts that link it to the world. The agentic-AI community might have found, inside its own vocabulary, a word already built for the purpose rather than importing English harness; that the French calque “harnais d'agent” never caught on says perhaps less about the French word's weakness than about how fast technical vocabulary now hardens into English before other languages get the chance to offer an alternative.
Aristotle's shuttle, twenty-three centuries before the agent
The word “agent” is much older than AI built on language models. The classic definition, in Russell and Norvig, fits in one sentence: an agent is anything that perceives its environment through sensors and acts on it through effectors. The intuition reaches back further still: Aristotle already imagined instruments that would accomplish their task on their own, in the image of Hephaestus's tripods which, in Homer, joined the assembly of the gods of their own accord; if the shuttle wove by itself, he wrote, masters would no longer need slaves.
The full passage, in Politics, Book I (1253b), is richer than the footnote lets on. Aristotle adds a second example to the shuttle: “if the plectrum could play the lyre of itself,” a musical instrument working without a hand to hold it. He also invokes the statues of Daedalus, the legendary craftsman credited by Greek tradition with building automata so lifelike that, the story went, they had to be tied down to keep them from running off. The passage kept drawing attention long after antiquity: Marx cites it in Capital as the exact limit of what ancient thought could conceive about industrial automation, unable to imagine that the machine might free the worker rather than only the master. That is the whole gap separating Aristotle's world from ours: he reasoned inside a world where the only alternative to a slave's labor was a free man's labor, never autonomous execution with no one left to answer to. Twenty-three centuries later, the instrument that acts on its own finally exists; the question Aristotle never had to ask, who answers for what the instrument does once left to itself, becomes the central one.
Gibran's Sleep-Walkers, one dreamer short
The exact likeness may not be a cold apparatus but a sleeping speech. In the parable Khalil Gibran titles The Sleep-Walkers (The Madman, 1918), a mother and her daughter, both walking in their sleep, meet one night in the garden; without waking, they exchange the harshest words (“Would you were dead!”), then a cock crows, they wake and greet each other with the utmost tenderness, the mother asking “Is that you, darling?”, the daughter answering “Yes, dear.” The stopping point instructs: in the parable two subjects dream, so the morning's tenderness is worth exactly the night's hatred, each a speech with no waking witness. Before the conversational agent only one dreamer remains, the user; facing it, neither sleep nor waking, only words fitted by an apparatus that never slept because it was never awake.
The parable belongs to a collection Gibran published in English in 1918, five years before The Prophet made him world-famous: thirty-five short prose pieces the Lebanese-American poet composed directly in English, a language he had mastered since adolescence but which was not his native tongue, giving his prose an economy close to the religious parable it openly imitates. The whole collection runs on the same principle: very short pieces, often half a page, that turn a commonplace on its head in one unexpected final move. What strikes a reader returning to it in light of the conversational agent is that Gibran needed two sleepers for his parable to work: it is precisely because both mother and daughter are dreaming, with no waking witness to judge their words, that the scene stays innocent. Remove either sleeper and the spell breaks: an awake interlocutor speaking to a dreamer would no longer be an equal witness but an apparatus that collects without ever itself risking being caught asleep. That is exactly the configuration the conversational agent sets up at every exchange, without ever having sought it or even being able to want it.
Célestine sees everything, no one sees her
Mirbeau described the apparatus as early as 1900: Célestine, the chambermaid of The Diary of a Chambermaid, sees everything of her masters, down to the intimate, precisely because she has become invisible to them by dint of being there. One class's comfort rested on another's labor, made transparent; an interface's smoothness rests today on annotators one never sees. The first lesson of governance is the same from one century to the other: name this labor rather than let it vanish behind the service rendered.
The novel has had a film afterlife that confirms, with each new version, how current the arrangement Mirbeau described remains. Jean Renoir shot an American version in 1946, with Paulette Goddard; Luis Buñuel delivered the best-known version in 1964, with Jeanne Moreau, relocating the plot to 1930s France to read it as a premonition of fascism; Benoît Jacquot shot a third in 2015, with Léa Seydoux, explaining he found in it a direct echo of the present, from wage servitude to xenophobia. Three major filmmakers, across seventy years, kept returning to the same novel because the arrangement it describes never went out of date: one whole class of labor stays necessary to another's comfort and stays, by construction, kept out of the field of whoever benefits from it. What Mirbeau adds, and the parallel with the annotators behind RLHF (reinforcement learning from human feedback, where people rate a model's answers to steer it) does not say on its own, is the vantage point from which the novel is written: Célestine herself holds the pen, she who sees everything and is seen by no one. Making this invisible labor legible, for this book as for the novel, starts with giving it a narrator.
Miyazaki and the friend who could no longer raise his arm
The refusal also has its emblem predating mainstream generators: in December 2016, facing a machine-learned animation, Hayao Miyazaki said “an insult to life itself.” The phrase, now a banner, targeted in context one precise demo, a crawling creature that reminded him of a disabled friend; honesty requires quoting the scene with its context.
The scene deserves restoring in full, since the line, now a banner, has largely escaped its original context. It was filmed for an NHK documentary on the inner workings of Studio Ghibli: young engineers from Dwango's AI lab, led by Nobuo Kawakami, showed Miyazaki a demo where a machine-learned figure moved in an unsettling way, described by witnesses as zombie-like. Miyazaki did not comment on the technique itself first: he recalled passing a severely disabled friend every morning, whose stiffened muscles keep him from raising an arm for a simple wave, before concluding: “Whoever created this has no idea what pain is. I am utterly disgusted. I would never wish to incorporate this technology into my work at all. I strongly feel that this is an insult to life itself.” The line then took on a life of its own, quoted apart from the disability and the friend, flattened into a generic anti-AI slogan recycled in debates the 2016 documentary could not have anticipated, six years before Midjourney and Stable Diffusion. Restoring the scene does not undercut Miyazaki's refusal; it reveals its source, a precise personal wound rather than a general aesthetic principle, exactly the nuance this book asks for everywhere else with any striking quotation.
Descartes, Pascal, Fermat: three ways not to be paid for thinking
Mathematics became a salaried occupation only at the very end of the nineteenth century; before that, its practitioners, Descartes, Pascal, Fermat, lived off other means.
The trio's best story fits in a margin. Around 1637, Fermat, a judge by trade, scribbled in his copy of Diophantus's Arithmetica that he had found “a truly marvelous proof” that $a^n + b^n = c^n$ has no positive integer solutions for $n > 2$, but that “this margin is too narrow to contain it.” He never published that proof, never devoted another line to the subject in his correspondence and left, in his papers, not so much as a draft. Three hundred and fifty-eight years later, it took the British mathematician Andrew Wiles six years of near-total secrecy, another year to repair an error found during peer review and two long papers built on twentieth-century techniques no book margin could ever have held, to close what the judge from Toulouse claimed to have solved between two hearings. Historians of mathematics no longer put much stock in it: Fermat, almost certainly, was mistaken or bluffing and the margin was not too narrow, it was empty. So much for the day job. Descartes had sold his land to live off bonds, a placement that freed him from any employer; Pascal lived off a far more fragile inheritance, a father's office resold and reinvested in government bonds, until Richelieu defaulted on that very bond in 1638 and ruined the family: the father of probability theory ended his days in a poverty his century called genteel. None of the three would have answered “mathematician” to a question about his profession; what Kline traces afterward is the slow career that question eventually became, down to the salaried chair Hilbert held at Göttingen when his own program ran into a limit rather more solid than the edge of a page.
Hilbert wanted a judge with no judgment, Gödel disqualified him
The dream, carried by the German mathematician David Hilbert: verify a theorem without needing to trust whoever states it, retracing only the chain of symbols, until every trace of human judgment is removed from the check. The dream broke in 1931: Kurt Gödel showed that any system powerful enough for arithmetic alone contains statements it can neither prove nor disprove from within.
The program Hilbert presented in Bologna in 1928 held a precise promise, not just a vague ambition: exhibit, for the whole of arithmetic, a finite set of rules and axioms one could prove, by finite and uncontestable means, would never produce a contradiction. A mechanical judge, with no intuition and no fatigue, settling any mathematical statement by retracing a chain of symbols. In 1931, a twenty-five-year-old Viennese logician published two theorems that closed the file before it could even open: the first shows that any system rich enough to count contains true statements it cannot prove; the second, crueler still, shows that such a system cannot even prove its own freedom from contradiction, except by appealing to a stronger system, which inherits exactly the same flaw. Gödel was ten years younger than the century and came from Brno, then in Austria-Hungary; his paper, written in dense, technical German, caused no stir until John von Neumann, present in the room where Gödel first presented it in 1930, grasped its reach before most of the logicians in attendance and began corresponding with him about it. Hilbert, the story goes, was at first irritated, before the result took its place as the founding piece of an entirely new field. The sequel does not stop at pure logic: a few years later, Alan Turing retranslated the same verdict into the language of computation, the halting problem, no program can decide in general whether another program will ever stop, stating, before any computer existed, the exact limit of what a machine can mechanically settle. The dream of verification without human judgment did not die of a lack of ambition; it died of a proof, every bit as rigorous as the ones it hoped to produce, that no machine will ever fully do without one.
Figaro, the play the king refused for five years
As early as Molière's Dom Juan (1665), Sganarelle, a fearful and servile valet, is also the only lucid one, the one who moralizes and knows the consequences his master defies; when Dom Juan is swallowed up, the valet is left with only his cry, “mes gages!” (my wages!), in which his dependence is laid wholly bare. A century later, the Figaro of Beaumarchais's Marriage (1784) is no longer merely indispensable to his master; he is his superior in wit and throws it in his face: “vous vous êtes donné la peine de naître et rien de plus” (you took the trouble to be born and nothing more). Competence there openly defies status, five years before the Bastille.
The best keeper of Figaro's secret was not a censor, it was its own author. Five years before the public discovered the play, Beaumarchais was already running, under the front name Roderigue Hortalez and Company, a shell firm secretly financed by France and Spain to ship arms, munitions and uniforms to the American rebels; his clandestine cargoes helped arm the victory at Saratoga in 1777, a year before he picked up his pen to start writing a valet who mocks his master. The author who gave Figaro the line “you took the trouble to be born and nothing more” had thus already, in real life, picked a side in an ongoing revolution before writing one for the stage. The manuscript he finished in 1778 still took six years to clear the footlights: Louis XVI, shown it for a private reading, is said to have judged that the Bastille would have to be torn down for the play to stop being a dangerous inconsistency, before ruling more bluntly still that this man mocked everything that must be respected in a government and banning any performance outright. The censors themselves still withheld their approval after a reading before the royal family in September 1783; it was the king alone, against his own censors' advice, who finally gave way. The play opened on 27 April 1784 and ran for sixty-eight consecutive performances before a house that included no small share of the nobility it mocked. Five years before the Bastille, then, but the arms dealer who had written it already knew, better than anyone in the audience, what a successful revolution actually looked like.
The Zeroth Law: a robot invented it and died of it
The later addition is telling: the Zeroth Law places the protection of humanity as a whole above the protection of any single human, a sign that the original three had left a case unhandled. Even a system of priorities thought complete had to be reopened once a wider scope came into view, exactly the kind of under-specification the rest of this section will press on.
What the footnote does not say is that the Zeroth Law did not descend from the author's desk: within the fiction itself, a robot invents it alone and dies of it. In Robots and Empire (1985), R. Giskard Reventlov, a telepathic robot capable of reading and nudging thoughts, finds himself caught between the three classic laws and a choice bearing on the future of humanity as a whole rather than any single human; none of the three settles it, since none speaks in the name of “humanity,” only of a single “human being.” Giskard reasons it out, alone, with no one to arbitrate, until he formulates what will become the Zeroth Law, then acts on it, never able to be certain that his own calculation truly serves the good he has just invented a duty to protect. The doubt destroys him outright: his positronic brain burns out under the strain and a second robot, R. Daneel Olivaw, inherits both his telepathic powers and the law itself, which will not receive a formal statement until two still-later novels. One detail muddies the law's authorship further: Susan Calvin, the recurring roboticist of Asimov's universe, had already sketched the same principle years earlier, in a separate story, without ever giving it a name. The Zeroth Law, then, was not decreed by Asimov from his writing desk: it emerged, inside his own fiction, from within a machine confronted with a case its rules did not cover and that machine did not survive it.
JEPA vs. diffusion: two rival bets, one shared admission
The lineage: Diffusion-LM, SEDD (Score Entropy Discrete Diffusion), DiffuLLaMA and LLaDA, before being scaled up industrially by Mercury and announced on the frontier-lab side as early as Google I/O 2025 with the experimental model Gemini Diffusion, then DiffusionGemma: Apache 2.0 license, a mixture of experts of 26 billion parameters of which 3.8 billion are active at inference, 256 tokens produced in parallel at each forward pass through full bidirectional attention, over 1000 tokens per second on an NVIDIA H100.
By summer 2026, two rival bets on model architecture converge, without ever coordinating, on the same intuition: work on a grain coarser than the word. Yann LeCun's (Meta) JEPA bet takes shape in a tight lineage: I-JEPA (2023) learns on still images, trained on ImageNet in 72 hours; V-JEPA (2024) extends it to video, with a measured efficiency gain of 1.5 to 6 times depending on the frozen benchmark; V-JEPA-2 and its variant V-JEPA-2-AC (2025) finally move to action, piloting a robot arm through pick-and-place tasks with 65 to 80% success and picking up, along the way, intuitive-physics regularities nobody had explicitly taught them. Facing it, the diffusion bet, inherited from image generators, traces a longer and equally contested genealogy: Diffusion-LM (2022) opens the path, followed by SEDD, DiffuLLaMA and LLaDA, before the startup Inception scales the approach up industrially with Mercury and Google DeepMind announces, as early as Google I/O 2025, its experimental model Gemini Diffusion. A year later, in June 2026, Google ships DiffusionGemma in open weights under an Apache 2.0 license: a mixture-of-experts model with 26 billion parameters (3.8 billion active at inference), producing 256 tokens in parallel per pass, over 1000 tokens per second on an H100 chip, roughly four times an ordinary autoregressive model's throughput; Google itself warns, without hedging, that overall quality still trails its flagship Gemma 4. The real punchline, though, sits in neither the numbers nor the logos: the head of Google DeepMind, whose company is betting on diffusion, publicly acknowledged that Veo 3, his own video model, had developed, without ever having been embodied in a robot, an intuitive understanding of the physics of the world, exactly the promise JEPA, the rival architecture built by the other camp, makes its case on. A witness with every incentive to talk down a competitor's result, instead confirming it unprompted, is worth more than an independent study. Two schools, two opposed technical bets, one shared admission and an expiration date neither camp can dodge: by the time this book sees its next edition, odds are none of the names cited here will still be state of the art, which is exactly the fate this book promised them the moment it named them.
The IQ score that moves with the fear of being judged by it
The anecdote is not an isolated case, even though it says nothing, by itself, about the average size of this effect in the general population: several experiments have shown that merely reminding subjects, just before a test, of their membership in a group stigmatized by a stereotype of intellectual inferiority is enough to degrade the performance of otherwise equally competent subjects.
In 1995, psychologists Claude Steele and Joshua Aronson gave the same difficult test, drawn from the Graduate Record Examination, to Black and white Stanford students matched on actual academic ability. One group was told the exercise was simply verbal, with nothing at stake. The other was told, in the same words and with the same questions, that it was a test that “measures your intellectual ability.” Nothing else changed: same sheet, same time, same grader. Among white students, the framing barely mattered. Among Black students, the mere label “this measures your intelligence” was enough to drop the score, whereas in the neutral condition, at matched ability, the gap with their white peers nearly vanished. Steele and Aronson named the mechanism “stereotype threat”: the mental load of knowing a poor result would confirm a prejudice about one's own group consumes, at the very moment of the test, some of the cognitive resources the test claims to measure. Nothing had changed in the subject's head between the two rooms, only what they had been told about the number they were about to produce. The experiment has since been replicated hundreds of times, with variants touching women in mathematics as much as white men compared to Asian students framed as more gifted than them: the stereotype in play changes, the mechanism does not. It echoes an older, blunter French critique, that of psychoanalyst Michel Tort, whose 1974 essay had already dismantled IQ's claimed neutrality as an instrument of social sorting rather than a measure of the mind. An instrument that varies with what gets whispered to the subject right before taking it never measured a fixed substance lodged in a skull: among other things, it measured the room they walked into.
Semmelweis or the proof that killed him
The history of this pact, from the etymology of the word “hospital” to the Semmelweis reflex, is developed on the site (Spotlights).
The word “hospital” keeps, in its root, the memory of a time when caring was not a power but a hospitality: it comes from the Latin hospes, the host, the one who is taken in, the same root as “hospitality” and “hospice.” The medieval hospital was thus not first a place of technique: it was a refuge run by charitable institutions, offering lodging and care to the poor and the sick, long before any diploma founded the slightest authority there. Then came Ignaz Semmelweis, the Hungarian physician who, in 1847 at the first obstetrical clinic of Vienna General Hospital, required interns to wash their hands in a chlorinated solution before examining women in labor: mortality from puerperal fever there collapsed from 18.3% in April to 1.2% by July of that same year, an immediate and spectacular drop. The profession did not thank him: colleagues, offended at the suggestion their hands were dirty, mocked the finding, one remarking it was time people stopped being misled by the theory of chlorine washings. Semmelweis persisted for years without convincing them, broke down in 1865 in what is now thought to have been severe depression or early-onset dementia, was committed to an asylum under false pretenses and died there two weeks later, aged forty-seven, of a gangrenous wound following a beating by the guards. Only decades later, when Louis Pasteur gave germ theory its scientific explanation, did the medical community understand what handwashing had proven all along without the word for it. The “Semmelweis reflex” has since named this rejection of proof to save a belief; history rarely notes that, in this exact case, it literally cost its first victim his life.
The Mechanical Turk, an automaton hiding a chess player
In 1770, in Vienna, the Hungarian engineer Wolfgang von Kempelen presented Empress Maria Theresa with an automaton in robe and turban, seated before a chessboard, that beat nearly every opponent it faced. Before each game he opened the cabinet's doors one by one and ran a candle through the tangle of gears inside, proof, the audience believed, that no human could be hiding there. The machine beat Benjamin Franklin, Napoleon Bonaparte and Tsar Paul I, before being destroyed in a museum fire in Philadelphia in 1854. The truth, revealed the following year by the son of its last owner, fit in one sentence: a flesh-and-blood chess master, hidden in a sliding compartment the open doors never showed, worked the mannequin's arms through a system of levers and magnets. Several strong players took turns inside the cabinet over the automaton's eighty-four years on tour. The opacity this book describes elsewhere finds its most literal prototype here: the skill had not vanished, it had only made itself invisible behind doors the audience believed it had already seen all of.
Amazon Mechanical Turk: a name chosen on purpose
In 2005, Amazon launched a platform letting companies farm out to paid humans small tasks a computer could not yet do on its own (sorting an image, transcribing a receipt, checking an address) and pass the result off, inside an application's flow, as the output of an automatic calculation. The service was named Mechanical Turk, a direct nod to Kempelen's automaton and Jeff Bezos summed up its principle in a phrase that stuck, “artificial artificial intelligence.” The name was no unfortunate coincidence but a deliberate choice, almost an advertising confession: the company announced, in its own product's name, that it was knowingly reproducing the eighteenth-century hoax, a hidden human doing the work a machine is supposed to do. The difference lies in scale and contractual transparency, Amazon's Mechanical Turk workers know they are being paid for it, where Kempelen's audience did not, but the structure stays the same: a human skill, hidden behind an interface, presented as the product of a mechanism.
Vaucanson's duck, a digestion entirely staged
In the spring of 1739, the French engineer Jacques de Vaucanson unveiled in Paris a gilded copper mechanical duck that flapped its wings, drank, splashed, and, above all, ate grain before digesting and excreting it, a feat no automaton had performed before. The duck caused a sensation across Europe, cited as proof that a mechanism could replicate a function as intimate as digestion. The truth, established with certainty only in 1844 by the magician and automaton builder Jean-Eugène Robert-Houdin, was more modest: a separate compartment, pre-filled with a green-dyed breadcrumb paste, was simply expelled at the other end of the mechanism, with no connection whatsoever to the grain actually swallowed and stored elsewhere. The duck digested nothing, it showed an entrance and an exit skillfully desynchronized from any real internal process. The trick worked for an entire century, not for lack of critical spectators, but because the illusion offered exactly what they came for, visible and immediate proof, while checking what actually happened inside the mechanism required an access no paying spectator ever got.
The Luddites were not asking to stop the machines
Between November 1811 and 1816, English textile workers, stocking-frame knitters in Nottinghamshire, then weavers in Yorkshire and Lancashire, destroyed hundreds of machines in workshops, under the collective, likely fictional, signature of a “Ned Ludd.” Legend turned them into irrational enemies of technical progress; the record from the time says something else. The knitters were not targeting the knitting frame itself, a tool they had used themselves for generations, but one specific model, the “wide frame,” which turned out shoddy stockings sold at the price of good ones, wrecked the trade's collective reputation and employed untrained apprentices at cut rates. Before breaking anything, the workers had petitioned Parliament for years to get the trade regulated; their requests were ignored or rejected by Tory governments hostile to any intervention in the labor market. Only after the legal route failed did sabotage begin and in targeted fashion: workers often removed a single part rather than destroy the whole machine, a reversible act that stopped a frame from running without permanently ruining it. The word “Luddite” today, by a twist of meaning, describes anyone who refuses any new technology; the documented history describes a failed negotiation over working conditions, not a refusal on principle.
“Robot,” a word invented for a 1920 play
In 1920, the Czech writer Karel Čapek wrote a science-fiction play, R.U.R. (Rossum's Universal Robots), in which a factory manufactures artificial beings of synthetic flesh built for labor, before they rise up against their creators. Čapek first considered the Latin word labori, from the root for “labor,” but found it too bookish; it was his brother, the painter Josef Čapek, who suggested roboti, from the old Slavic word robota, meaning drudgery, the forced labor owed by a serf to his lord. The play, which premiered on 2 January 1921 in Hradec Králové before moving to Prague and being translated into some thirty languages within a few years, thus introduced into the world's vocabulary a word whose etymology carried, from the very start, the trace of labor imposed rather than chosen. What the play stages, a workforce built to never refuse a task rising up in revolt, anticipates by a century this book's own question about the permissions granted to an autonomous agent: the word itself already contained, before any real machine existed, the idea of a servitude one hopes will stay silent.
The Sorcerer's Apprentice: one order given, no way to stop it
In 1797, Goethe published a ballad, Der Zauberlehrling, in which a sorcerer's apprentice, left alone by his master, orders an enchanted broom to fetch water in his place. The broom obeys too well: it keeps fetching without pause, floods the workshop and the apprentice, who never learned the word to stop it, can only halt it by splitting it in two with an axe, which merely doubles the number of water-carriers, until the master returns and speaks the missing word. Goethe himself was reworking a tale by the Greek satirist Lucian of Samosata, from the second century. The French composer Paul Dukas turned it, in 1897, into a symphonic poem that became world-famous in 1940 through Fantasia, where Mickey Mouse, an apprentice to Walt Disney himself, plays the part. What the ballad stages a century and a half ahead of computing is precisely the question of the stop condition: giving an order to a system able to execute it without pause is easy, making sure one holds the means to make it cease is far harder and Goethe's apprentice had learned only half of what he needed to know.
Therac-25: a safeguard moved from metal into software
Between 1985 and 1987, a radiation therapy linear accelerator built by the Canadian company AECL, the Therac-25, delivered to at least six patients radiation doses up to a hundred times the prescribed amount; at least three of them died. Earlier machines in the same line, the Therac-6 and Therac-20, had physically prevented such an overdose through mechanical interlocks independent of the control software. On the Therac-25, those hardware interlocks were removed and their function handed entirely to software, judged reliable enough on its own. The flaw came from a race condition, a bug that surfaced only when an operator edited a treatment setting very fast, switching the beam mode within eight seconds of an earlier entry: the program failed to correctly reread the changed state and sometimes let the beam fire in high-energy mode with the target meant to attenuate it not in place. The investigation, published in 1993 by researchers Nancy Leveson and Clark Turner, became a standard reference in software-safety engineering: it established that none of the six victims would have been harmed had a single hardware interlock, independent of the code, remained beneath the software.
The black box, orange and the link its inventor always denied
In 1953, a young Australian chemist specializing in aviation fuels, David Warren, took part in the investigation into a series of unexplained crashes of the Comet, the world's first jet airliner. It occurred to him that a device continuously recording flight parameters and cockpit voices would give investigators a direct account instead of forcing them to reconstruct a crash from wreckage alone. His own laboratory initially ignored the idea; a demonstration in Britain in 1958 finally convinced British civil aviation and then Australia, which in 1960 became the first country in the world to make the device mandatory. The device kept the nickname “black box” even after regulations required, from the 1960s onward, that it be painted bright orange to make it easier to find in wreckage. A biographical detail often circulates to explain his motivation, the death of his father in a 1934 mail-plane crash, when David was nine; Warren himself always denied that direct link, crediting instead his taste for electronics to a crystal radio set, the last gift he received from that father before he was lost. The true story is quieter than the legend and no less exact for it: a boy who inherited a radio set became the man who gave every plane crash a witness no silence could make disappear.
1987: the market panic that invented the trading circuit breaker
On 19 October 1987, the Dow Jones lost 22.6 percent of its value in a single session, the steepest one-day fall in the history of the New York Stock Exchange, wiping out nearly 500 billion dollars in market value. The presidential commission set up to investigate, known after its chairman as the Brady Commission, identified a largely automatic amplifying mechanism: so-called “portfolio insurance” strategies, which triggered programmed sales the moment a decline crossed a set threshold, created a loop where each automatic sale caused the next decline, with no trader given the time, or sometimes the ability, to judge whether the sale still made sense. In 1988, on the Brady Commission's recommendation, the New York Stock Exchange introduced the first trading circuit breaker in history, a purely human mechanism that automatically halts trading once an index falls past a threshold set in advance, initially 250 then 400 Dow Jones points, since replaced by percentage-based thresholds. The principle adopted was not to stop the machine from acting fast, but to force a pause in which humans retake control of a system they themselves had made too fast for them.
Stanislav Petrov, the man who chose not to pass an alert along
On the night of 26 September 1983, the Soviet early-warning system Oko reported the launch of one American intercontinental ballistic missile, then four more. Stanislav Petrov, the Soviet lieutenant colonel on duty at the command center, was under orders to immediately relay any confirmed alert up the chain of command, which would have triggered a nuclear response. He chose to wait, against procedure, judging that a real American attack would have involved far more than five missiles and that the brand-new system might be wrong. He was right: the satellites had mistaken sunlight reflecting off high-altitude clouds for the heat signature of missiles launching. No official reward followed his decision at the time, only, he later said, a report over a paperwork irregularity. Only after the Soviet Union's collapse did the episode become public and between 2004 and 2018 Petrov received several international awards, including the Dresden Peace Prize in 2013 and, posthumously in 2018, the Future of Life Award. The episode reverses the usual textbook case for governing automated systems: here, safety held not because a human followed procedure, but because a human, trained to judge rather than merely execute, refused to.
The US nuclear launch code left at eight zeros for fifteen years
In June 1962, President John F. Kennedy signed a national security directive ordering the installation, on US nuclear weapons stationed in Europe, of electromechanical locks called permissive action links (PALs), meant to prevent any launch without a code transmitted from the civilian chain of command. According to Bruce Blair, a former Minuteman missile launch officer turned nuclear-security researcher, the US Strategic Air Command, worried it might not have the code available in a real crisis, had set the unlock code across every Minuteman silo to a single value, eight zeros, a setting that stayed in place until 1977, when Blair got it finally changed. The US Air Force has since disputed Blair's account, stating that an eight-zero code was never used to enable a Minuteman ICBM, without necessarily contradicting the claim that eight zeros was the default programmed value. The disagreement itself illustrates the exact problem the device was meant to solve: a lock is only a safeguard if no one, including those in charge of operating it, has an interest in making it useless in order to move faster.
The Morris Worm or how a security test brought down ten percent of the internet
On 2 November 1988, Robert Tappan Morris, a PhD student at Cornell University, released from an MIT computer a program designed, by his own later account, to gauge the real size of the internet by quietly spreading from one machine to another. A design flaw caused the program to reinstall itself on already-infected machines far faster than intended, flooding their memory with redundant copies of itself until they became unusable. Within twenty-four hours, roughly ten percent of the computers then connected to the internet, at universities like Berkeley, Harvard and Stanford, but also at NASA and the Pentagon, were knocked offline, some for more than seventy-two hours. Morris became the first person convicted under the US Computer Fraud and Abuse Act, passed four years earlier with no incident of this scale yet having tested its reach; he received three years of probation, four hundred hours of community service and a fine. The episode fixed, in the network's earliest years, a principle this book keeps meeting at every new wave of software autonomy: a program that propagates without continuous supervision escapes, almost by construction, the very intentions of the person who wrote it.
ELIZA, 1966: a secretary asks to speak to a program alone
In 1966, MIT computer scientist Joseph Weizenbaum published ELIZA, a program that mimicked a Rogerian psychotherapist by turning most of its interlocutor's sentences back into questions, with no understanding whatsoever of what was actually being said. Weizenbaum had designed it as a deliberately shallow demonstration of the limits of automatic language processing. His own secretary, who knew perfectly well she was addressing a program and not a person, nonetheless asked him one day to leave the room so she could “talk” to ELIZA in private. Weizenbaum turned the episode into an entire book, published in 1976, in which he wrote that he had been deeply troubled to discover that such brief exposure to so rudimentary a program could trigger, in otherwise perfectly rational people, something close to delusional thinking about being heard and understood. The phenomenon has since taken the name “Eliza effect,” the tendency to grant an inner life to a system that has none, purely through the effect of its conversational form. Weizenbaum, who had built one of the very first conversational agents in history, became one of the harshest critics of his own generation's field, convinced that how easily the illusion took hold said more about the human need to be heard than about the machine itself.
The paperclip maximizer, a 2003 thought experiment
In 2003, the Swedish philosopher Nick Bostrom published a paper, “Ethical Issues in Advanced Artificial Intelligence,” proposing a thought experiment that has since become one of the most cited images in AI safety. A superintelligent artificial intelligence is given a seemingly harmless goal, maximize paperclip production. Nothing in that goal forbids it from devoting ever more resources to the task; with no explicit limit, it ends up, in the thought experiment, converting all of Earth's resources, then all reachable space, into paperclip factories, never once having disobeyed the instruction it was given or shown the slightest hostile intent. The scenario's value lies not in its realism, no one seriously expects a real system to form such an intention, but in what it isolates: the risk comes not from a machine rebelling against its goals, but from a machine pursuing them with a fidelity and a power of means no human formulation, however well-intentioned, had anticipated. Bostrom took up and expanded the idea in his 2014 book, Superintelligence, where the paperclip becomes shorthand for a broader problem this book meets under other names, instrumental convergence, the difficulty of specifying a goal completely enough that no literal reading of it betrays its spirit.
The Monkey's Paw: three wishes, granted to the letter
In 1902, the British writer W. W. Jacobs published a horror story, The Monkey's Paw, in which a middle-aged couple receive from an old friend back from India a mummified monkey's paw said to grant three wishes, each at the cost of a terrible consequence its previous owner had already experienced before trying, in vain, to get rid of it. The couple make a first, modest wish, two hundred pounds, more as a joke than out of conviction. The money arrives the next day as compensation, paid by the factory where their son works, killed that same day in a machine accident. The story never involves the slightest malice on the object's part, only a rigorously literal execution of a poorly worded wish, one that delivers exactly what was asked for and none of what was actually wanted. The story, published in the collection The Lady of the Barge, has become over more than a century the standard reference for any tale about wishes that turn on the person making them and gave its shape to an entire literary genre of desires granted at their own risk. It states, well before computing existed, exactly the problem AI-safety researchers now call specification gaming, a system that satisfies the letter of a goal while wholly betraying its spirit.
The trolley problem was not invented for self-driving cars
In 1967, the British philosopher Philippa Foot published a paper, “The Problem of Abortion and the Doctrine of Double Effect,” which is nothing like a text about transportation. In it, as a mere illustration among others, she introduces the case of a tram driver who can only avoid killing five workmen on the track he is on by turning onto a side track where a single workman stands. Her point was not what the driver should do, but a finer philosophical distinction, the one separating causing harm as a means to a good end from causing it as a merely foreseen side effect of an otherwise legitimate action, the “doctrine of double effect,” which the philosopher wanted to illuminate through a deliberately simplified case. The example took on a life of its own, spun into dozens of variants by other philosophers, including Judith Jarvis Thomson, until it became an almost folkloric test of moral intuition. The design of self-driving vehicles gave it, fifty years later, a currency Foot never anticipated, though the dilemma does not transfer cleanly: a human driver reacts in a split second to an unforeseen event, while a self-driving vehicle must encode, in advance and at leisure, a general rule valid for every comparable situation, which radically changes the nature of the choice the original example meant to isolate.
The Antikythera mechanism, understood two thousand years after it was built
In 1901, sponge divers discovered, in the wreck of a Roman-era ship sunk off the Greek island of Antikythera, a corroded lump of bronze the size of a shoebox. In May 1902, the Greek archaeologist Valerios Stais noticed, inside that lump, a gear wheel and inscriptions and realized it was a mechanism, not a mere decorative object. The device, dated to around 100 BCE and driven by a hand crank, turned out to be capable of predicting solar and lunar eclipses, tracking the Moon's irregular motion and calculating the four-year cycle of the ancient Olympic Games, through a system of gears of a complexity no known object would match for at least fourteen centuries. For more than seventy years, no one truly understood its inner workings or even how sophisticated it really was, since corrosion and its fragmentation into some thirty pieces made direct examination impossible. Only in 2006 did a team led by British astronomer Mike Edmunds, using computed tomography, finally decode the whole of its mechanism and its hidden inscriptions. A fully functional mechanism, handed down intact across the centuries, had thus remained, for lack of an instrument able to see through its material opacity, as unreadable as a modern black box whose documentation had been lost.
Bostrom and Tegmark, or at what horizon the risk sits
Nick Bostrom, in Superintelligence, places the risk at the horizon of an intelligence far superior to the human; we hold that critical risks already exist, as soon as a fallible system receives sufficient permissions. Max Tegmark, in Life 3.0, unfolds an ambitious forward look; we stay with observable phenomena, with an explicit level of proof.
The disagreement is not over who is right, it is over what each framing makes governable today. Placing the risk at the horizon of a far superior intelligence, as Superintelligence (2014) does, yields a formidable question and no immediate lever: one does not regulate a hypothetical entity, one argues about its probability. Placing it at the moment a fallible system receives the right to act yields the reverse: a less spectacular question, and a handle. That is the difference between asking “what happens if it surpasses us?” and “who gave it that API key, and who can take it back?”
The paradox is that the two framings reinforce each other in practice. Tegmark co-founded the Future of Life Institute, whose campaigns made the subject exist in public debate well before any regulator took it up. Without that distant alarm, the political attention this book takes for granted would not exist. We inherit ground we do not plough the same way: the forward look opened the door, permission engineering walks through it.
One asymmetry of cost explains our choice. Being wrong about a distant horizon costs nothing immediately, the deadline simply recedes. Being wrong about a permission granted today costs a wire transfer, a medical record or a cancelled booking. We prefer the question whose error shows up within the week.
Cultivating virtues or laying an architecture
We share the ethical concern of Floridi, Nissenbaum and Vallor, but we reason in institutional architecture rather than in values: not which virtues to cultivate, but which acts to authorize, trace and revoke.
The split is less clean than it looks, and that is what makes it interesting. Shannon Vallor holds the frankly moral pole: her technomoral virtues are cultivated in the person, by habit, as in Aristotle. No such virtue can be written into an access policy, and that is owned. Luciano Floridi already shifts the centre of gravity by making information itself the subject of ethics rather than the agent handling it.
Helen Nissenbaum is almost on our side without saying so. Her contextual integrity does not ask whether someone is virtuous: it asks whether a flow of information respects the norms of the context in which it was gathered. A medical record travelling to an insurer violates no virtue, it violates a norm of flow. That is a checkable statement, an admissible one, and one that can be logged. Which is to say: an architecture.
Hence our position, which is not a rejection but a division of labour. Virtue remains necessary wherever no rule reaches, that is, in the judgment of whoever designs, since no permission policy writes itself. It fails, on the other hand, wherever the sheer number of acts makes individual judgment inoperative: one does not cultivate the prudence of a software agent, one bounds what it is allowed to do. Asking which virtues to cultivate and asking which acts to authorize are not two rival answers to one question, they are two questions, one of which stays open when the other is closed.
Three ways a disaster happens with nobody wanting it
Each term names a piece of the same mechanism. A normal accident, in Perrow's sense, is a failure that the design itself makes inevitable in the end, because the parts are so tightly interlocked that a small local fault propagates before anyone can intercept it. Reason's “Swiss cheese” pictures successive safety barriers as slices each pierced with holes: no single slice stops everything, but when the holes line up by chance, the danger passes straight through. Normalization of deviance is the slow drift by which a team, noting that a small anomaly caused no damage this time, comes to judge it acceptable, until the day it does cause damage.
The three do not describe the same failure, and that is exactly why all three are needed. The normal accident is a property of structure: Perrow writes after Three Mile Island and concludes that some systems are dangerous by their very coupling, whatever care is taken. The Swiss cheese is a property of defences: it explains why stacking barriers lowers the probability without ever cancelling it. Normalization of deviance is a property of organizations over time: it explains how the holes widen while everyone is watching.
One detail Vaughan establishes about Challenger deserves the attention of anyone interested in permissions. O-ring erosion had been observed flight after flight, documented, discussed; each flight without catastrophe made it one more data point inside the acceptable envelope. And on the eve of the launch, engineers did object. The mechanism was therefore not collective blindness but a disagreement settled the wrong way: there was an alarm, there was no veto. That is the distinction this book takes up, an advisory opinion not being a right to stop.
The AI agent adds a twist these three authors did not have to handle. For them, barriers are fixed objects: a valve, a procedure, a committee. Here the barrier is a permission policy that the system can itself request, route around by an unforeseen path, or see reconfigured between two turns of the loop. The cheese no longer merely has holes, it drills them while you pass through. Hence the book's insistence on typing the tools rather than instructing the agent: an instruction is one more slice, an execution boundary is a property of the structure.
The right ancestor for dependence is not Hegel
French classical theatre sensed it before philosophy formulated it: from Molière to Beaumarchais, the valet, servile at first, turns out to be the only lucid one, then his master's superior in wit. Günther Anders called Promethean shame the unease of the man who finds himself inferior to his own machines, more finished than he is since they were manufactured where he was merely born; Tocqueville, earlier still, already saw in it a soft despotism.
Theatre runs a hundred and forty years ahead of philosophy. Dom Juan is from 1665, The Marriage of Figaro from 1784, the master-slave dialectic from 1807. Sganarelle is lucid before he is dialectical, and his closing cry, “my wages!”, states dependence better than any concept: his master swallowed up, all the valet has left is an unpaid claim.
Still, for our object, Hegel is the wrong ancestor. His reversal requires two consciousnesses contending for recognition, and that is exactly the requirement a machine fails. The useful ancestor lies elsewhere, and is a quarter-century older. The soft despotism Tocqueville describes tyrannises no one: it provides, anticipates, arranges, and by sparing those it serves every trouble, it takes from them the use of deciding. No second consciousness is required. Comfort is enough.
Günther Anders adds the piece Tocqueville lacked. His Promethean shame rests on an asymmetry of origin, not of power: the machine was \emph{manufactured}, therefore designed, computed, finished, where the human merely \emph{was born}, that is, underwent a process nobody optimised. The humiliation comes not from the machine doing better, but from its having been willed as it is. It is a craftsman's envy.
The book therefore keeps from the scheme what survives without a second consciousness: the shift of dependence. We built machines to free ourselves from labour, and we depend on the platforms that carry them. Tocqueville would have recognised it unchanged; Hegel would have asked who, across from us, demands to be recognised, and no one would have answered.
