27 NOVEMBER 2019
In a recent episode of the dystopian anthology series Black Mirror – spoiler alert! – the British police are called to a live crime scene. On the line are executives at Smithereen, an addictive American social media platform caught up in the situation. At every step, the firm turns out to know more about the perpetrator than the authorities of his sovereign country. The officers are public-spirited and decent, and the criminal is not a straightforward baddie or lunatic, but the sovereignties of both state and individual are overwhelmed by the reach of a Californian data giant that knows his darkest traumas, can listen in to him, and can plumb its algorithms to understand his emotional make-up.
The episode could be a dramatisation of The Age of Surveillance Capitalism, the celebrated book by the American social psychologist Shoshana Zuboff. Seven years in the making, it anatomises the rise of “instrumentarian power”, whereby firms capture data about human behaviour, process it and turn the resulting information advantage into an instrument of prediction and control. It is the tale of a new Gilded Age in which giant firms – real-life Smithereens such as Facebook and Google – have unfair and undemocratic power over states and individuals. Zuboff’s thesis, like the Black Mirror episode, leaves you briefly thinking: if only the state could wield the same data power as the top-knotted Silicon Valley billionaires. But then images of overreach and authoritarianism swim before your eyes and you think: oh God, that would be hell.
It falls not to Black Mirror to offer a glimpse of that other extreme but to journalists, because a world where the state wields even more data power than Facebook or Google already exists. It is Xinjiang, a mountainous region of north-west China and home to 11 million Uighurs, a mostly Muslim central Asian people. On 24 November the International Consortium of Investigative Journalists (ICIJ) provided that glimpse, in the form of an unprecedented leak of secret Chinese government documents.
The cache sheds light on a network of government camps – the largest internment of an ethno-religious minority since the Second World War – that can hold more than a million people. It exposes the Integrated Joint Operations Platform (IJOP), the surveillance programme that decides who gets interned. The platform collects information from social media, other websites and apps, facial recognition cameras, health data and checkpoints, and uses it to identify citizens who might threaten China’s one-party leadership and ethnically Han Chinese-dominated society. Things as banal as prayer and foreign travel can tip the balance. These people are often interned as “students”, subjected to dehumanising treatment and allowed to “graduate” back into society only after they have undergone an “ideological transformation”. The leaks show that in one week in June 2017 the IJOP flagged 24,412 “suspicious persons”, 15,683 of whom were rounded up. The system’s arbitrariness, driven by algorithms based partly on artificial intelligence (AI), infantilises the population and creates a paranoid atmosphere.
You might write this off as but a ghastly peculiarity of China’s techno-autocracy. Even if that were the case, there would be a strong argument for sanctions on officials and firms involved in the Xinjiang nightmare. But it is not. Xinjiang is a Petri dish for technologies and methods that can be applied elsewhere. Governments and firms from around the world got to marvel at the emotion recognition systems pioneered in the region at China’s main security trade fair in the city of Shenzhen last month. And while Google or Facebook may not cooperate with Western governments to the extent that their Chinese counterparts do with Beijing, China shows how naturally Big Tech and government can rub along together; the gulf between an omniscient Smithereen and the clueless British police portrayed in Black Mirror may come to seem quaint. Zuboff has rightly argued that the West cannot be complacent about the elision of instrumentarian and authoritarian power, warning of “a weakened democracy with a totalitarian leader who wants to grab control… It happened in the 20th century and can happen again.” A big-data Weimar, in other words.
There is, however, hope in Zuboff’s thesis. She notes that the original Gilded Age of monopolistic oil and railroad firms in the 19th century was tamed in the 20th century by new rules and institutions: trust-busting regulations, trades unions and eventually the New Deal welfare state (or social democracy, as it’s called in Europe). One can extend her comparison: if Silicon Valley billionaires are the Rockefellers of the data age then China’s leaders are the new Lenins, melding the power of data with that of the state to create an alternative, more abusively centralised model. Once more, in the early decades of a century human civilisation is faced with a grim choice between domination by private firms and domination by overmighty states. Yet once more, there is a healthier third way.
Today, that is some form of data social democracy. States should be able to harness and tame the power of data and AI, but with clear controls and limits. New institutions must arise to mediate between interest groups. Civil society needs to step up (the campaign for the EU’s new data privacy regime, the GDPR, is a good model). Most of all, sovereign citizens must hold sway over the new forms of power and be able, collectively, to curb monopolies. So where the call once was “neither Rockefeller nor Lenin” and “neither Washington nor Moscow”, now let it be “neither surveillance capitalism nor techno-autocracy” and “neither Silicon Valley nor Xinjiang”. Pluralism and democracy must win the day. Because – spoiler alert! – the alternative is nightmarish.