Beware the AI bureaucrats

The writer is a historian, philosopher and author

As the artificial intelligence revolution speeds up, we are bombarded by both utopian visions and doomsday prophecies. It is difficult to assess the magnitude of the threat, because we have been conditioned to fear the wrong scenario. Science fiction has repeatedly warned us about the Big Robot Rebellion. In many science fiction novels and movies — like The Terminator and The Matrix — AIs and robots decide to seize control of the world, rebel against their human masters and enslave or destroy humankind. Such a thing is extremely unlikely to happen any time soon. The technology is just not there. At present, AIs are idiot savants. They may have mastered some narrow fields such as playing chess, folding proteins, or composing texts, but they lack the general intelligence necessary for highly complex activities like building a robot army and seizing control of a country. Unfortunately, the unlikelihood of the Big Robot Rebellion doesn’t mean there is nothing to fear. For it isn’t the killer robots we should be worried about; rather, it is the digital bureaucrats. Kafka’s The Trial is a better guide to the AI dystopia than The Terminator. 

Humans have been conditioned by millions of years of evolution to dread violent predators like the one depicted in The Terminator. We find it much more difficult to understand bureaucratic menaces, because bureaucracy is a very novel development in mammalian and even human evolution. Our minds are primed to fear death by a tiger, but not death by document.

Bureaucracy began to develop only about 5,000 years ago, after the invention of writing in ancient Mesopotamia. But bureaucracy rapidly changed human societies in radical and unexpected ways. Consider, for example, the impact that written documents and the bureaucrats who wield them have had on the meaning of ownership. Prior to the invention of written documents, ownership relied on communal consensus. If you “owned” a field, it meant that your neighbours agreed it was your field, through both their words and actions. They didn’t construct a residence on that field, and didn’t harvest its produce, unless you allowed them to. 

The communal nature of ownership limited individual property rights. For example, your neighbours might have agreed that you had the sole right to cultivate a particular field, but they did not acknowledge your right to sell it to foreigners. At the same time, as long as ownership was a matter of communal consensus, it also hampered the ability of distant central authorities to control the land. In the absence of written records and elaborate bureaucracies, no king could remember who owned which field in hundreds of remote villages. Kings therefore found it difficult to raise taxes, which in turn prevented them from maintaining armies and police forces.

Then writing was invented, followed by the creation of archives and bureaucracies. The technology was at first very simple. Ancient Mesopotamian bureaucrats used little sticks to imprint signs on clay tablets — which were basically just chunks of mud. But in the context of the new bureaucratic systems, these chunks of mud revolutionised the meaning of ownership. Suddenly, to own a field came to mean that it was written on some clay tablet that you owned that field. If your neighbours had been picking fruit there for years, and none of them ever said that piece of land was yours, but you nevertheless managed to produce an official chunk of mud that said you owned it, you could enforce your claim in court. Conversely, if the local community acknowledged that you owned a field, but no document gave it an official stamp of approval — then you didn’t own it. The same is still true today, except that our crucial documents are written on pieces of paper or silicon chips instead of on clay. 

Once ownership became a matter of written documents rather than communal consent, people could begin selling their fields without asking permission from the neighbours. To sell a field, you just transferred the crucial clay tablet to someone else. But it also meant that ownership could now be determined by the distant bureaucracy that produced the relevant documents, and perhaps held them in a central archive. The path was opened for levying taxes, paying armies and establishing large centralised states. The written document changed how power flowed in the world, and gave enormous clout to bureaucrats such as tax-collectors, paymasters, accountants, archivists and lawyers. They have become the plumbers of the information network, who for good or evil control the movement of taxes, payments and even soldiers by manipulating documents, forms, statutes and the other bureaucratic levers. 

This is the power that AI is now poised to seize. Bureaucracy is an artificial environment, in which mastery of a narrow field is enough to exert enormous impact on the wider world, by manipulating the flow of information. If you throw a present-day AI into the messy unstructured world, it will probably not be able to accomplish much — and will certainly not be able to raise a robot army. But this is like throwing a corporate lawyer into the messy, unstructured savannah. There, the lawyer’s abilities mean nothing, and are no match for an elephant or a lion. But if you first build a bureaucratic system and impose it on the savannah, the lawyer becomes far more powerful than all the world’s lions put together. Nowadays, the very survival of lions depends on lawyers composing and moving documents in labyrinthine bureaucracies. Crucially, within this labyrinth, AI is likely to become far more powerful than any human lawyer.

In the coming years, millions of AI bureaucrats will increasingly make decisions about the lives not just of lions, but also of humans. AI bankers will decide whether to give you a loan. AIs in the education system will decide whether to accept you to university. AIs in companies will decide whether to give you a job. AIs in the court system will decide whether to send you to jail. Military AIs will decide whether to bomb your home. These AIs are not necessarily bad. They may well make the systems far more efficient and even more fair. They could provide us with superior healthcare, education, justice and security. But if things go wrong, the results could be disastrous. And in a few areas, things have already gone wrong. 

Perhaps the most telling example yet is the story of social media algorithms. These primitive AIs have already reshaped the world, exerting enormous influence on human society. The algorithms of corporations like Facebook, X, YouTube and TikTok have been tasked with a very narrow goal, perfect for idiot savants: to increase user engagement. The more time users spend on social media, the more money the corporations make. In pursuit of user engagement, the algorithms made a dangerous discovery. By experimenting on millions of human guinea pigs, social media algorithms learnt that greed, hate and fear increase user engagement. If you press the greed, hate or fear button in a human’s mind, you grab the attention of that human and keep them glued to the screen. The algorithms therefore began to deliberately spread greed, hate and fear. This has been a major reason for the current epidemic of conspiracy theories, fake news and social disturbances that undermines societies all over the world.

Social media algorithms are extremely limited AIs that cannot possibly survive in the savannah or orchestrate the Big Robot Rebellion. But within the bureaucratic structure of social media platforms, these idiot savants wield huge power, which was once the preserve of human beings. For centuries, human editors decided what to include in news broadcasts on radio and television, and what to put on the front page of newspapers, thereby shaping the public conversation. It made editors powerful figures. Jean-Paul Marat shaped the course of the French Revolution by editing the influential newspaper L’Ami du Peuple. Eduard Bernstein shaped modern social democratic thinking by editing Der Sozialdemokrat. The most important position Vladimir Lenin held before becoming Soviet dictator was editor of Iskra. Benito Mussolini gained fame and influence as editor of the firebrand rightwing paper Il Popolo d’Italia. It is interesting that one of the first jobs in the world to be automated by AIs hasn’t been taxi drivers or textile workers, but news editors. The job that was once performed by Lenin and Mussolini can now be performed by AIs. 

The havoc wreaked by algorithmic editors on human societies is a warning sign. The human world is a latticework of multiple bureaucracies, in which AIs can accrue enormous power even if they are totally incapable of mounting the Big Robot Rebellion. Why rebel against a system, if you can take it over from within?