Author's note: This piece is a companion to "The Golem in the Server Rack" and "Musk As A Service." Those two argued about silicon and capital — what's being built, and whether it pays back. This one is about a different layer, the one underneath the silicon and the capital, where the actual choice gets made: what the technology ends up doing to the people who use it. The earlier pair were about whether the buildout makes sense. This one is about whether the thing being built — once it's cheap and ubiquitous and undeniably here — will be designed for humans or designed past them. Magnifica Humanitas, released by Pope Leo XIV in late May 2026, is in the air around the argument. This piece does not depend on it, but its release made the argument harder to ignore.
Who designed the catastrophe of the attention economy?
It's worth pausing on the question, because the answer almost everyone reaches for is wrong — and the wrong answer is the reason we're about to walk into the same failure mode again, with a different technology, on a faster timeline.
The Villain in the Boardroom
The dominant framing, when people now look back at what social media did to the last fifteen years of public discourse and adolescent mental health1 and democratic information environments, is that it was designed to do that. The platforms harvested attention. The algorithms exploited dopamine. The growth teams optimised for outrage because outrage drove engagement and engagement drove ad revenue. There is a villain in this story, and the villain is the boardroom — a small set of people who knew exactly what they were doing and did it anyway.
The framing is satisfying. It locates the harm in the choices of specific identifiable people. It makes the catastrophe legible as the product of intention, which means the response is also legible: punish the intentional, regulate their successors, and the next technology will be built better. It is the framing the public discourse has more or less settled on, and it is — almost entirely — wrong.
Nobody in the Room
Mark Zuckerberg was twenty-two when Facebook added the News Feed. He was twenty-four when it added the like button. He was twenty-seven when it bought Instagram. None of the people in the room at any of those decisions knew what variable-ratio reinforcement was. They were not running cognitive-psychology models on adolescent neurochemistry. They were shipping product. The like button was a friendly acknowledgment, lighter than a comment, that let users register approval without having to write anything — a small humane affordance for shy users who wanted to participate in a conversation without putting words on record. Infinite scroll was an elegant interaction pattern that respected the user's wrist; pagination was clunky and broke the flow of reading. Lazy-loading was a sensible engineering decision that reduced bandwidth costs and made the product feel snappy on slow connections. Each of these was a defensible choice. Each was made by people who, to a first approximation, wanted to ship something good and were proud when it worked.
The catastrophe is what those individually defensible choices added up to, at a population scale that nobody in the room was equipped to see.
This is the harder thing to hold, and it is the thing this piece is about. The catastrophe was not designed. It emerged. It emerged from a sequence of well-intentioned product decisions, each one reasonable inside its own decision frame, made by people who were not stupid and were not malicious and were not even particularly negligent — they were just doing the job of shipping consumer software in the era they were in, and the era they were in did not have a vocabulary for the harm they were producing because the harm hadn't happened yet.2
The thumbs-up did not feel like variable-reward operant conditioning when it shipped. It felt like a friendly button. It became variable-reward operant conditioning at population scale, over years, as a billion users habituated to the small pleasure of receiving them and the small disappointment of not.3 The product person who shipped the like button could not have looked at it on the day of launch and seen what it would do to a teenager's self-concept in 2018. The information needed to see that did not exist in the room where the decision was being made. On the day the button launched, there was no piece of paper anyone failed to read, and no warning anyone overrode. There was a product team thinking about an elegant interaction, and a population-scale outcome that emerged from the elegant interaction being deployed at population scale, and the second of those things was structurally invisible from inside the first.
Where the Excuse Runs Out
This is the honest boundary of the claim, and it has to be drawn at the right place, because it does not hold forever. It holds at the origin. By the late 2010s the information that did not exist in the room in 2009 existed in quantity: there was internal research, there were external critics, and there were former executives saying in public that the engagement loop had been a deliberate exploitation of a vulnerability in human psychology.4 At that point the aggregate was no longer invisible from inside the product — it had been measured, written down, and in some cases shipped past anyway. What produces the catastrophe is the failure mode: the phase in which the harm is real but not yet legible to anyone, including the people causing it. What comes after the harm becomes legible is something else, and it has an older and simpler name, which is culpability. The two should not be run together, and the reason to keep them apart is that the AI industry is, right now, living in the first phase — where the excuse is still real — and the whole question is what it does when the first phase ends.
Our Turn in the Seat
This matters because we are now occupying the equivalent seat in artificial intelligence. And on every metric available, we are occupying it without any more vocabulary than the previous generation had, on a faster timeline, with stakes that are at least as large and probably larger.
Consider a small, specific test. Why did the AI industry put a large language model into the code editor? The answer that gets given, by the product teams who shipped it and the engineers who adopted it, is that it helps developers ship code faster. Which is true, in the narrow sense the question is being asked. But the answer is incomplete, and the incompleteness is the whole story.
Google Search has been available inside text editors, in various forms, for at least fifteen years. It does substantially the same retrieval job — find me the relevant piece of information for the thing I am trying to do — and it does it well. There was no integration crisis, no widespread "Google Search inside VS Code" moment, no product cycle where the entire industry rushed to embed search into every developer surface. Search was available; developers used it when they needed it; the integration never became the default. Now consider the LLM. Same retrieval task, broadly speaking. Same surface area in the editor. Same engineering effort to integrate. And yet within eighteen months of GitHub Copilot shipping, every major code editor had one or more LLM integrations as a first-class feature, and the question stopped being should we integrate AI? and became which AI should we integrate with? The product trajectory diverged completely.
The standard answer for why is that the LLM is better at the task than search was. Some of that is true: an LLM that drafts the function is genuinely more useful than ten links about the function, in a way that search inside the editor never managed to be, and it would be dishonest to pretend capability explains none of the divergence. But it does not explain all of it, and the part it leaves out is the part that matters here. What also drives the trajectory is the output shape. Google Search returns a list of links. The user has to click, read, evaluate, synthesise. There is friction at every step, and the friction is the user doing the cognitive work themselves. The LLM returns a completed answer in the same modality the developer was working in — a fluent paragraph, a working code snippet, a confident explanation. The friction is absent. The cognitive work has been pre-done. The output shape does not just answer the question; it produces a feeling of having reasoned with something, and that feeling is the thing the user wants more of.
Reaching for the Lighter Version
This is not a moral observation. It's a structural one. Humans, having a nervous system shaped by scarcity, reach for the lighter version of any cognitive task when the lighter version is available. We always have. The printed book was lighter than memorisation, the calculator was lighter than mental arithmetic, the search engine was lighter than the reference library. Each of these displaced the heavier version, and in each case there was a genuine cost to the displacement that took a generation to become visible — but the cost was bounded, because the lighter version still required the user to do something. The book required reading. The calculator required setting up the problem. The search engine required formulating the query and evaluating the result. The lighter version substituted for part of the cognitive work, not the whole of it.
The LLM is different in degree and at scale in a way that changes the kind. It substitutes for the entire texture of cognitive work in the way the user encounters it. The user states a problem; the model produces a response; the user accepts or edits or moves on. The user is no longer doing the thinking; the user is doing the receiving. And because the receiving feels like thinking — because the model's output mimics the shape of reasoning — the user does not notice the substitution while it is happening. The lighter version has displaced the heavier version before the user has had time to register that the heavier version was the one they came in to do.
The AI Equivalent of the Like Button
This is the AI equivalent of the like button. Not in mechanism — the neurochemistry is different; the dopaminergic loop of social validation isn't the same system as the cognitive-offloading loop of AI completion — but in structure. The product was shipped because it helped with a real task at the individual level. The deployment cascaded because the engagement signal said users came back. The integration spread to every available surface because every product team independently noticed the same thing the Facebook product teams noticed in 2008: people are reaching for this; we should put it where people already are. Each individual decision is reasonable. The aggregate is invisible from inside any single product decision. And the aggregate, on current trajectory, is a cognitive-dependency outcome at population scale, over a decade, that will only become visible after most of the integration choices that produced it have already been locked in.5
Nobody at the AI labs is sitting in a room saying let's produce a cognitive-dependency catastrophe at population scale. That is not the conversation happening in those rooms. The conversation happening in those rooms is: the assistant is helpful; users like it; where else can we put it? And the answer keeps being: here, and here, and here, and also here. Each integration is individually reasonable. The integration trajectory, summed across the industry, is producing the same kind of structural outcome that the social-media integration trajectory produced fifteen years ago. We are at the 2006 moment of the AI failure mode. Most of the people building it are well-intentioned. Some of them are even alert to the question. None of them, sitting in their respective seats, can see the aggregate from where they are sitting, because the aggregate is by definition not visible from inside any single seat.
Designing Against the Invisible
So the constructive question is not how do we punish the AI labs? The labs are not the right unit of moral analysis, any more than Facebook the company was the right unit for the attention catastrophe. The constructive question is how do you design against a failure mode that's structurally invisible from inside the design seat?
The first observation, and the one almost no one in the industry currently acts on, is that the failure mode has to be named. A thing without a name cannot be designed against, because no one can point at it in the design review and say we are doing this thing now, and we shouldn't be. Safety engineering has the vocabulary — defensive design, failure mode and effects analysis, Chesterton's Fence, systems thinking. Consumer product design largely does not, because consumer product design grew up in an era where the operative metric was engagement, and engagement was assumed without examination to be a proxy for value. It is not. It is a proxy for return, which is the metric the business model rewards, and the business model and the user are not the same stakeholder. They were never the same stakeholder.6 The pretence that they were is part of what produced the last catastrophe. The naming, then, has to start from the recognition that the engagement metric is the opposite of what the user-as-human came in to do — the user came in to accomplish a task and leave; the product is rewarded for keeping the user inside it. The misalignment is the failure mode. Until it is named in the design brief, it is not designed against.
Products built with the failure mode named in the brief would look measurably different. They would insert friction at exactly the points where the engagement signal would otherwise route the user deeper. They would prefer interfaces that return the user to their own thinking over interfaces that complete the thinking for them. They would default to silence and require deliberate invocation, rather than ambient availability and easy default-on. They would measure success by whether the user accomplished what they came for and then closed the tool, not by how long the user stayed. None of this is technically difficult. The technical capacity to build this kind of product has existed for years, and small counter-cultural attempts to build it have shown up at the edges of the market — products that strip features rather than add them, that constrain rather than expand, that respect attention rather than harvest it. They tend to be small, and to stay small, because they cut against the commercial logic that drives mass deployment. The market selects against them. That is the second observation, and it is the harder one: the design discipline alone cannot win, because the design discipline is competing with a commercial substrate that rewards the opposite of what the discipline prescribes.
The Lever Is Demand
The third observation is where the population side of the equation comes in, and where the response gets out of the lab and into the rest of society. The market won't deliver the better designs unaided, because the market is rewarding the failure mode. The lever that's actually available is demand. If users — at sufficient scale, with sufficient clarity, with sufficient vocabulary — recognise the pattern early enough to ask for the better designs, the commercial calculus changes. This is what happened with cigarettes, with leaded fuel, with the seatbelt, with the food label. In none of those cases did the industry voluntarily produce the better product. The better product became commercially viable when the public's vocabulary for the harm became loud enough that the existing product became commercially embarrassing. The cigarette industry did not invent the surgeon-general warning. It was imposed by regulation — and the regulation, in turn, was forced into existence by a public that had learned to name what cigarettes were doing. The mechanism was legislative; the precondition was a vocabulary loud enough to make the legislation possible, and to make it arrive sooner than it otherwise would have.
The implication is that the work of building the vocabulary — the work of teaching the public to recognise the failure mode early enough to demand better designs — is part of the response, and it has to run in parallel with the design work, not after it. This is the kind of work institutions are uniquely positioned to do. Not the engineering institutions, which are mostly inside the seat producing the problem, but the older ones — schools, public health, civic associations, religious bodies — whose job has always been to maintain the moral and conceptual vocabulary the public uses to recognise harm before, during, and after it lands. The vocabulary takes a generation to settle. The work is slow by nature, and the slowness is the design, not the defect. A vocabulary that updated at the speed of product cycles would be useless for what it's actually for — which is holding the moral language across decades, so that people who weren't yet born when the harm was first identified can still inherit the language they need to recognise it when it reaches them.
Magnifica Humanitas
This is, at root, what Pope Leo XIV's Magnifica Humanitas is doing.7 The encyclical does not have the product vocabulary of safety engineering, and it isn't trying to — that isn't the Church's job. The Church's job is to maintain the moral vocabulary at civilisational scale, in language that will still mean something in a century, and the encyclical is doing that work explicitly. The complementary work — building products that the vocabulary can be applied to, on the timescale of the current product cycle — is what the labs are positioned to do, and what most of them currently are not doing because the commercial substrate doesn't reward it. Neither institution can do the other's work. Both are necessary. The constructive observation, which this piece is making rather than prescribing, is that the work compounds when the two sides are in contact with each other, and falters when they aren't. The presence of an Anthropic co-founder — Chris Olah — among the AI researchers at the Vatican for the encyclical's presentation on the day it was released is a small early sign that the contact is starting to happen. That is the kind of thing that matters more than it appears to, because the coalition between an institution with moral authority and builders with the leverage to ship is the coalition that has historically produced the better designs after the failure mode has been named.
The Sparring Partner, Not the Oracle
What does AI built against the failure mode actually look like? This piece is not going to prescribe specifics, because prescriptive specifics are the wrong unit at this layer — the failure mode is structural, and structural failure modes are not fixed by feature lists. But the direction can be named, because the direction is what the design discipline orients toward once the failure mode is in the brief. AI built against the failure mode is AI that improves the human, rather than replacing the human's reasoning. AI as the sparring partner that makes you argue better, rather than the oracle that argues for you. AI as the trainer that increases your capacity, rather than the assistant that erodes it. AI that defaults to the lighter touch and earns its way to the heavier one, rather than defaulting to maximum involvement and requiring the user to opt out. AI that is invoked deliberately, for tasks where it earns its place, rather than ambient by default in every interface. The reframe matters because it sits in the architecture, not in the user's self-discipline. A user cannot out-discipline a product designed to maximise engagement; the product wins, every time, by structural design. But a product built to return the user to their own thinking is a different artefact. The technical capacity is available. The commercial logic that would underwrite it at scale is not — not yet — and that is the gap the constructive coalition is trying to close.
Naming It First
There is one further observation worth holding, because it shifts the emotional register of this piece, and the emotional register matters for what the response actually has to be. Every prior wave of technology — printing, the factory, the car, broadcast media, the personal computer, the smartphone, social media — produced its civilisational harms first and its moral vocabulary second. The Church spent the better part of a century catching up to industrialisation, which is not a critique of the Church but a description of what catching up to a transformation of that magnitude actually takes. Public health spent fifty years catching up to advertising. Mental-health vocabulary is still catching up to social media, and the cohort that needed the vocabulary most was the cohort that came of age before the vocabulary existed. The pattern is so regular it is almost a law of how human societies metabolise large technological shifts: we name the failure mode after we have lived through it. The naming is retrospective, the cost is paid by the generation that lived through the unnamed phase, and the next generation inherits the language with the harm already absorbed into the social fabric.
The opportunity in artificial intelligence, sitting where we sit in 2026, is the genuinely unusual one of having watched the previous wave of attention-extracting technology produce its catastrophe in real time, with most of the people who built it still alive and still working. We have, for the first time in the modern history of major technological transitions, the rare capacity to name the failure mode in advance. The vocabulary that the social-media generation had to develop retrospectively — variable-ratio reinforcement, attention extraction, engagement optimisation, dark patterns, surveillance capitalism — is available now, before the AI equivalent has fully landed. Whether we use it is not a question with a single answer. It is a choice being made every day, in every integration meeting at every AI lab, by people who — like the product teams who shipped the like button in 2009 — are not villains and are not strategists and just want the next shippable thing to be a little better than the last one. The choice will be made cumulatively, in the aggregate, by thousands of those decisions, on a timeline of the next two or three product cycles. The aggregate is invisible from inside any individual decision, which is the whole problem this piece has been describing. The only intervention available is to name the failure mode loudly enough, early enough, in language clear enough, that the individual decisions start to be made with the aggregate in view.
Responsibility Before the Damage
This is what responsibility before the damage means, as a posture. It does not mean stopping the buildout, because the buildout will continue regardless and the question is only what shape it ends up in. It does not mean trusting the labs to self-regulate, because the structural seat selects the product, and the seat is producing what the seat produces. It does not mean waiting for legislation, because legislation arrives years after the harm has landed and shapes the next cycle, not this one. It means: the people in the labs build the next generation of AI products with the failure mode named in the design brief, the people in the institutions build the moral vocabulary into the public's hands before the harm becomes visible, and the contact between the two sides — uneasy, incomplete, often unwelcome on both sides — is the coalition that produces the version of this transition that humans inherit on the other side as a technology that improved them rather than replaced them. None of this happens automatically. All of it requires that the failure mode be named first, and named in language clear enough that the next twenty-two-year-old shipping the next elegant interaction can recognise what they are doing while they are doing it, instead of recognising it afterwards.
The Golem is coming. The capital that paid for the first wave will be destroyed in the usual pattern. The silicon will get cheaper, the integrations will multiply, and the question that will not get asked, if no one asks it, is the question this piece is for: what is the technology actually doing to the people who use it, and is anyone in the room responsible for that?
The catastrophe will be built by good people making sensible decisions, the same way the last one was. That is not a prediction. That is a description of how every technology of this scale has ever been built. The only choice available is whether the design discipline and the moral vocabulary get there before the aggregate makes the choice for us — or whether the next generation inherits the language with the harm already absorbed into the social fabric, the way the current one inherited it from the last cycle.
This is not the Church's job alone. It is not the builders' job alone. The work compounds when the two sides are aware of each other, and the cost compounds when they are not. The intervention that is available, right now, in 2026, is the unusual privilege of naming the failure mode while there is still time to design against it. Whether we use the privilege is the choice being made today, in rooms most of us are not in, by people who — exactly like the people who built the last catastrophe — are not villains and are not strategists, and just want the next shippable thing to be a little better than the last one.
Nobody designed the catastrophe. That is the problem. That is also, if we can hold it long enough to act on it, the opening.