AI Serves to reduce the biopolitical subject into mere data that can be dominated and controlled.

In theoretical terms, we witnessed the discourse transforming from biopolitical to technological determinist – foregrounding technology as the cause underlying social phenomena, good or bad, and de-emphasising concerns about institutionalised racism and mass surveillance (see also, Gangadharan and Niklas 2019). Ironically, this shift was precipitated by a well-intentioned scientific experiment carried out by a civil rights group. It is possible the ACLU experiment helped the bill get past the assembly floor by making the harms FR can cause appear more concrete and measurable. However, it simultaneously reduced the concerns with FR to the level of the technology itself – an example of what Selbst et al. (2019) have called a failure of abstraction. The social discourse lost its biopolitical magnitude – and so did the bill.

Shahin (2019) has drawn attention to the theoretical significance of ‘critical junctures’ – emergent conditions in which a social discourse takes on a new direction without the principal stakeholders intending it to – in shaping regulations about technology. The ACLU experiment in August 2019 was such a critical juncture in the discourse about AB-1215. Even though ACLU’s own position on why FR had no place in law enforcement did not shift after the experiment – its press releases, for instance, retained their focus on systemic issues – both Assembly member Ting and the media latched on to Rekognition’s manifest failure as a piece of technology. This quickly undermined the original intent of the bill. To be sure, other factors may have also played a role in the willingness shown by Ting and other supporters of the bill to accept a sunset clause – twice – and change the character of the bill from a permanent ban on the use of FR in police body cameras to a three-year moratorium. But the noticeable change in the social discourse following the experiment, coinciding with changes in the bill itself, does indicate that the experiment weakened the bill even as it became instrumental for its passage.

Our analysis is significant not only for future research on FR but also for future efforts to check algorithmic governance, legislative or otherwise, in the US and around the world. Firstly, it underlines the significance of a biopolitical approach to understanding – and resisting – algorithmic governance. That does not mean technological flaws are not important to point out. But those flaws are themselves the consequence rather than the cause of institutionalised racism: they don’t produce but serve to re-produce discrimination and marginalisation.

Research and resistance therefore need to press forward with an agenda in which FR and other forms of artificial intelligence are viewed as sociotechnical artefacts interpellated in relations of power – produced by them even as they serve to reproduce those relations. Moreover, a technologically determinist outlook, where activists focus only on the machine, its algorithms, input and output, and not as much on the social contexts of its design and use, is not only a failure of abstraction (Selbst et al. 2019) – it is also a failure of strategy. Blaming the technology alone might appear attractive in the short-term but, as our analysis indicates, it does not aid the long-term goal of regulating algorithmic governance as a means of achieving social justice.

Secondly, these relations of power increasingly have both universal and particular – or global and local – dimensions. Being sensitive to such hybridity is important for research on and resistance to algorithmic governance in countries like the US, as our analysis indicates, but even more so in the Global South. That is partly because sociotechnical artefacts such as FR are constructed in North America and Western Europe, along with certain norms and practices of governance, and are often then ‘localized’ (Zaugg 2019) in the Global South amidst different forms of social hierarchy.

Understanding these hybridised dynamics opens new avenues for research and resistance aimed at exposing and destabilising such hierarchies. For instance, how are algorithms trained to discriminate against people of colour implicated in the biopolitics of societies where the entire population is ‘of colour’? If algorithms for governance are coded and trained from scratch rather than copied and pasted, what kinds of power and norms of control do they reflect and reproduce? Comparative research across countries, and empirical research focusing on specific countries and contexts of design and use of these technologies are important. As new legal instruments are developed to regulate this technology, collaborations between legal scholars and scholars of social science are key in understanding how we can negotiate these technologies as a society.

In conclusion, we emphasise the main argument of our study. Scholars and activists have long been aware of the role of algorithms and artificial intelligence in marginalising minorities and immigrants and reinforcing relations of power (boyd and Crawford 2012; Eubanks 2018). Biometric technologies, such as FR, are particularly insidious examples as they can act at the level of both the ‘body’ and the ‘body politic’. They reduce human beings into data points that may be stored, manipulated and controlled en masse (Browne 2010; Hood 2020). At the same time, they enable forms of domination that are systemic in nature: they have a long history and they are institutionalised in a variety of social practices. Indeed, algorithms themselves represent one such social practice. The prejudices they exhibit are a consequence of the systemic bias they are produced by – even as the algorithms help re-enact and reproduce that bias. Research on and resistance to algorithmic governance should, therefore, avoid the trap of technological determinism and not lose sight of the systemic nature of their subject matter – the biopolitics of discrimination and domination.