The Gatekeepers Need More Time

Slowing AI development helps incumbent institutions preserve artificial scarcity

This post was drafted and iteratively refined by ChatGPT in conversation with KMO¹. For posts written in KMO’s voice, see Gen X Science Fiction and Futurism.

KMO is on Medicaid now. After years of living without any medical coverage, necessity has prompted him to return to institutional medicine. His right shoulder is seriously messed up. He doesn’t know the exact nature of the damage. An MRI might reveal what an x-ray could not, but before Medicaid will consider paying for advanced imaging, he has to complete a course of physical therapy.

He has attended one session. The physical therapist measured his range of motion and showed him several exercises. None of this was useless, but the exercises are readily available on YouTube. On his way out, KMO received an invoice showing that the hospital had charged Medicaid $400 for the appointment.

Medicaid may ultimately pay much less than the amount billed. That does not change the function of the appointment. KMO is not simply receiving physical therapy. He is completing an institutional requirement that may eventually make him eligible for an MRI.

He expects to attend around a dozen sessions before the gatekeepers will consider approving the imaging. The therapist may provide some benefit along the way, but the process also creates a record showing that KMO has completed the required steps.

The scarce resource is not the MRI. It is permission.

The American medical system contains organizations with different and often conflicting interests. Government agencies, pharmaceutical companies, insurers, hospital systems and professional guilds do not operate as a single coordinated body. But each benefits from keeping medical care expensive, restricted and dependent on authorized intermediaries.

Professional guilds benefit from limits on who may practice medicine. Hospitals benefit from opaque prices and limited competition. Pharmaceutical companies benefit from patents and market exclusivity. Insurers benefit from controlling access to payment. Government agencies accumulate authority by deciding which treatments, providers and patients qualify.

Nobody needs to organize a conspiracy. Each participant protects its own position. Together they form an emergent medical cartel that restricts access and preserves artificial scarcity.

Artificial intelligence threatens this arrangement because much of the expertise used to justify medical gatekeeping may soon be cheap and widely available. An AI system could evaluate symptoms, measure a patient’s movement through a phone camera, recommend exercises, interpret medical images and explain its conclusions in ordinary language.

That does not mean the patient would be allowed to act on those conclusions. An AI might determine that KMO needs an MRI while remaining legally unable to order one. Medicaid could still refuse to pay. The hospital could still refuse to perform the scan. Professional guilds could still insist that only licensed practitioners are qualified to interpret the result.

Technical abundance does not eliminate institutional scarcity.

This is where AI alignment becomes more than a problem of making models obedient. An AI aligned with a hospital could help the hospital bill more efficiently. An AI aligned with an insurer or Medicaid administrator could enforce eligibility rules more efficiently. An AI aligned with a professional guild could preserve the requirement that patients pass through licensed intermediaries.

These systems might work exactly as their operators intended. By KMO’s definition, they would still be misaligned.

An AI aligned with institutions that depend on artificial scarcity is aligned against the people excluded by it.

The people defending those arrangements will not usually describe themselves as defending wealth, power or status. Most will never examine their motives that directly. They will gravitate toward explanations that portray their interests as enlightened principles.

Medical guilds defend standards and patient safety. Universities defend intellectual rigor. Copyright holders defend creativity. Regulators defend responsible innovation. Workers whose status depends on scarce cognitive labor defend human meaning.

Every gatekeeper has a moral vocabulary for explaining why the gate must remain closed.

Kirwin Hampshire provided a recent example in his essay about the future of mathematics. He worried that AI could deprive future mathematicians of the opportunity to make original discoveries and reduce them to spectators of machine-generated mathematics. He presented this as a threat to the spiritual dimension of mathematics.

There is something about mathematical discovery (progress, advancement, creation) which is vital to the spiritual, experiential quality of doing mathematics. The creation (or even the pursuit) of novel mathematics is one way that humans have historically accessed the ineffable and encountered the divine and mystical.

The spiritual concern may be sincere. It also dresses an obvious status anxiety in elevated language. AI threatens to deprive mathematicians of the distinction attached to discovering something before anyone else. Hampshire converts that threatened loss of professional and historical status into a threatened spiritual loss for humanity.

He is likely sincere. Sincerity does not cancel self-interest.

Medical advice can be dangerous. Intellectual standards can matter. Human mathematical activity can remain valuable. But these truths do not justify preserving every institution that claims to protect them. A legitimate concern can still provide moral cover for a threatened monopoly.

This bears directly on the question of how quickly AI should advance.

Slow development gives incumbent institutions time to absorb each improvement. They can create licensing requirements, designate approved providers, prohibit unauthorized uses and incorporate AI into existing systems without surrendering their positions. They can train models to treat institutional compliance as safety and call the result responsible alignment.

Governmental agencies and large bureaucracies move slowly. Frontier AI development does not have to move slowly with them. If capabilities advance faster than institutions can respond, the difference between what AI can do and what people are permitted to do with it may become too large to preserve through ordinary regulatory adaptation.

Rapid development does not guarantee a fair distribution of wealth or power. It does not guarantee that the institutions replacing the current ones will be better. It does, however, give current incumbents less time to preserve their positions by incorporating each new capability into the existing system of licenses, permissions and artificial scarcity.

Slow AI is easier for existing institutions to capture.

This is one reason KMO is skeptical of efforts to slow frontier development. Slowing AI may be presented as the cautious and humane option. It may also give the people who benefit most from current arrangements the time they need to ensure that AI buttresses their status.

The question is not whether AI will obey human institutions. The question is whether it will serve human well-being when human institutions stand in the way.

1. But if you find a typo or spelling error, then obviously that’s someplace where KMO tinkered with the text directly.