AI inevitably reinforces systems of discrimination – using datasets from discriminatory practices and differentiating individuals based on identities it leads to discrimination especially for governments in the Global South who can’t fight back against Northern tech companies
Arun 19 (Chinmayi, Affiliate at Harvard at the Berkman Klein Center for Internet and Society & Fellow of the Information Society Project at Yale Law School, 07-20-19, “AI and the Global South: Designing for Other Worlds” Draft Chapter for the Oxford Handbook of Ethics of AI, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3403010, pp. 9-12)-qcl
It is worth reading work by scholars who think about AI and discrimination, while noting that Southern institutions and legal frameworks can exacerbate the harms that they discuss. Southern populations within Northern countries might not have same access as privileged people to the institutions within the same countries. Autonomous systems are used so broadly that they can affect the economy, housing, intimate relationships and more. They can introduce or enhance discrimination and oppression, and they can erase populations by failing to account for their existence. I begin with discussing autonomous systems as systems of discrimination, I then move on to discussing what this may mean for Southern populations, especially since the fragile democracies and non-democracies of the world do not offer their citizens the institutional protections that may be available in the USA or Europe. Any discussion of AI in the context of discrimination has to discuss big data, which is ‘the fuel that runs the Algorithmic Society’. Algorithmic systems are often trained on a corpus of data, which means that the big data and its inherent biases affect the outcome of these systems.61 There are several stages at which inaccuracies and bias can be introduced into algorithmic decisionmaking. These range from the recording of the data to the actual question answered by the algorithm. There is a tendency to accept predictions based on datasets as the truth62 even though the the outcome is typically an interpretation of the data63 and may be inaccurate.64 The dataset could suffer from any number of problems which would skew the outcome. Scholars use the term ‘dirty data’ to refer to missing, incorrect and badly represented data, as well as to data that has been manipulated intentionally or distorted by biases.65 Crawford has pointed out that “not all data is created or even collected equally”.66 Data collection has embedded power and assumptions. The recording of fingerprints for example is difficult for those who do manual work such as refugees, and migrant and contract labourers.67 The very design of data sets can be biased as a result of assumptions and gaps.68 The datasets could under-represent or wrongly represent certain populations, leading to discrimination against them or to their exclusion.69 Even if the dataset is accurate, its structure can end up discriminating and marginalising people: the classic example being datasets that code people as either male or female, erasing other forms of gender identity.70 A dataset might discriminate indirectly by recording a seemingly innocuous fact that acts as a marker for identity. An illustration of this is employment which can be used to infer caste based on the historic employment of marginalised caste people for certain tasks (such as manual scavenging).71 The training data for algorithms can embed bias,72and algorithms trained on real world data would replicate real word discrimination.73 Therefore a hospital computer program used to sort out medical school students based on previous admissions decisions, ends up discriminating against women and racial minorities because of the rules it learned from the hospital’s older biased decisions. 74 Big data essentially generates correlations. 75 Although scientists understand the difference between correlation and causation, the rest of the world tends to treat conclusions based on big data as ‘enough’.76 The AI Now institute has articulated the problem in unambiguous terms.77 It has pointed out that since classification, differentiation and ranking are central to AI systems, these systems are ‘systems of discrimination’. It has argued that the bias in AI systems is connected with the lack of diversity in the AI industry, including the people who build AI tools and the environment in which they are built. The large scale AI systems come from elite university labs and a few technology companies, which are ‘white, affluent, technically oriented and male’ spaces.78 In other words, these technologies are designed by people from the North. Context can be reintroduced if universities studying AI collaborate with social and humanities disciplines, affected communities and civil society organisations. 79 It is important in to account for plurality, context and intersectionality.80 In addition to changing how decisions are made about design, data and deployment in the algorithmic society, we must give Southern populations the tools to engage effectively with the questions that affect them. This is already proving challenging in what we understand as Global North countries despite the lively debate and relatively strong privacy and anti-discrimination laws. When companies deploy these technologies in Southern countries, there are fewer resources and institutions to help protect marginalised people’s rights. This needs to be remedied as a high priority The systems discussed in the four case studies are designed by people with privileged access to the data of data subjects. The data subjects have little control or autonomy over their own data. It is typical, when autonomous systems are used, that the data subject has no idea who has access to their data or how it is used. 81 This is exacerbated in Southern countries. Young democracies lack institutional stability since it takes time to build institutions and institutionalise democratic practices.82 This is why Milan argues that we need diverse ways for citizens and civil society engagement to ward off datafication practices that result in oppression and inequality.83 The institutional frameworks of Southern countries must be taken into account as we consider what impact AI might have on the South. Freedom depends not just on political and civil rights, but also on other social and economic arrangement such as education and health care. 84 Development, Amartya Sen argues, depends on the removal of sources ‘unfreedom’ such as systematic social deprivation, poverty, poor economic opportunities and tyranny, Sen describes poverty in terms of capability deprivation, in what is now famously knows as the ‘capabilities approach’ to development. Julie Cohen has applied Sen’s work, as build on by Martha Nussbaum, to access to knowledge, and has pointed out that we need to pay more attention to the relationship between the networked information environment and human flourishing.85 The rights of Southern populations can be realised through efforts made by states, but can also be eroded by the governing elite of states. In the past, Southern countries worked together as a bloc, to gain access to technology, capital and markets. 86 They had a shared commitment to development, opposition of colonialism, the creation of equitable conditions for socio-economic development of all countries and the evolution of South-South co operation. 87This co-operation has been taking place since the Non Aligned movement, in which developing countries came together to negotiate development and trade issues. As the developing countries began what they called South-South co-operation, triangular co-operation also began such that donors and northern partners became involved in South-South initiatives.88 Progress has been made over the years on South-South initiatives but one might argue that the cooperation between the Southern states and triangular co-operation has had mixed results. Over the years, non State actors such as businesses and civil society have started playing a powerful role in Southern Countries. These countries have developed groups that are wealthy and influential, and populations that are more affluent than their fellow citizens - the extractive, exploitative consequences are evident in the Aadhaar case study. Some Southern states are more developed and have greater economic influence than other Southern states. The exploitative nature of this relationship is evident in the China-Zimbabwe case study.