The aff use of empirical and “unbias” data -- Big data is undermined with racist and ableist assumptions
Gillborn et al 18 (David Gillborn, Paul Warmington & Sean Demack (2018) QuantCrit: education, policy, ‘Big Data’ and principles for a critical race theory of statistics, Race Ethnicity and Education, 21:2, 158-179, DOI: 10.1080/13613324.2017.1377417, https://doi.org/10.1080/13613324.2017.1377417 ) // RB
Policy-makers, the media, and many academics treat quantitative material as if it is fundamentally different and superior to qualitative data. Numbers are assumed to report ‘the facts’; they are seen as authoritative, neutral, dispassionate, and objective. Indeed, governments do not use numbers merely to describe the world, they increasingly use statistics as an essential part of the technology by which they seek to re/shape educational systems. In this way, numbers play a key role in how inequality is shaped, legitimized, and protected. This has been called ‘policy as numbers’ (Rose 1999; Ozga and Lingard 2007; Rizvi and Lingard 2010): neo-liberalism has enhanced the significance of numbers and statistics as technologies of governance, as central to what Power (1997) calls the rise of the ‘audit society’ and what Neave (1998) has called ‘the evaluative state’. (Lingard 2011, 359)1 Numbers are increasingly used to justify policy priorities and to label teachers, schools, districts, and even entire countries, as educational successes and failures. National testing programs, such as the No Child Left Behind (NCLB) reforms in the US and the use of school performance tables in England, have popularized the idea that numbers can be used to expose (and change) failing schools (Gillborn and Youdell 2000; Darling-Hammond 2007; Barber 2012). For example, across the globe politicians and pressure-groups frequently try to make their case by quoting results from PISA (Program of International Student Assessment) – which is run by the Organization for Economic Co-operation and Development (OECD). Prominent examples exist in the States, the UK and Australia (see Lingard, Creagh, and Vass 2012). Countries’ positions in the PISA tables are often cited as if they unambiguously and accurately represent the relative quality of schooling in different nations (despite their very different populations and education systems). And yet the commentaries rarely include any detail about the relatively small samples (less than 200 schools in all but one of the US returns since 2000)(NCES n.d.); the selective curricular coverage of the tests (in reading, math, and science); nor the fact that students in different countries sometimes take different assessments or miss certain assessments altogether (Stewart 2013). Despite these severe limitations, the UK government frequently cites PISA results as evidence of the need for change (cf. Department for Education (DfE) 2015, 8) and has stated that it will ‘measure the increased performance of the school system as a whole by reference to international tables of student attainment, such as PISA’ (quoted in Scott 2016). Compare the confident use of PISA (below), by the then-Secretary of State for Education Michael Gove, and the more circumspect view offered by an academic critic: Since the 1990s our performance in these league tables has been at best, stagnant, at worst declining. In the latest results we are 21st amongst 65 participants in the world for science, 23rd for reading and 26th for mathematics. For all the well-intentioned efforts of past governments we are still falling further behind the best-performing school systems in the world. (Gove 2013) ‘There are very few things you can summarise with a number and yet Pisa claims to be able to capture a country’s entire education system in just three of them. It can’t be possible.’ Dr Hugh Morrison, Queen’s University Belfast (quoted in Stewart 2013).
Numbers and accountability On both sides of the Atlantic, policy-makers have argued that statistics will allow greater ‘accountability’ in education. But the thinking behind such claims is flawed in numerous ways. As Linda Darling-Hammond (2007) has noted, for example, under NCLB the numerous wider structural inequities that shape educational outcomes are ignored by focusing attention at the school level: … the wealthiest US public schools spend at least 10 times more than the poorest schools … Although the Act orders schools to ensure that 100% of students test at levels identified as ‘proficient’ … the small per-pupil dollar allocation the law makes to schools serving low-income students is well under 10% of schools’ total spending, far too little to correct these conditions (247–8) Additionally, the use of quantitative measures as a form of accountability assumes that the measures are valid, that is, that the recorded data bear some relevance to the issue/s that lie behind the targets. But there is often scope for cheating and some high-profile cases have emerged. In England, for example, documented cases include teachers altering students’ work and a school that removed low-attaining students from its official roll in advance of high-stakes testing, thereby artificially raising the proportion of students deemed ‘successful’ (Harding 2015). In the US, David Hursh notes that gaming the system can produce considerable rewards: Rodney Paige, as superintendent of the Houston Independent School District (and later chosen to be President [GW] Bush’s first Secretary of Education) … [ordered] principals to not list a student as dropping out but as having left for another school or some reason other than dropping out. Such creative book-keeping resulted in the district claiming a greatly reduced dropout rate of 1.5% in 2001–02 and winning a national award for excellence (Hursh 2007, 302) Numbers and equity In the UK, government policy puts numbers at the heart of its proclaimed strategy to create a fairer society. The Conservative Party, which formed the dominant partner in the Coalition Government (2010–2015), went into the 2010 general election with arguments about ‘transparency’ threaded throughout their Party Manifesto. This included the promise, emphasized as a bold subheading, to ‘Publish data so the public can hold government to account’ (Conservative Conservative Party 2010, 69). Subsequently, the rhetoric was translated into a policy that envisaged ‘the public’ using statistics to understand, challenge and then change the behavior of public authorities, including the Government itself: ‘Our proposals,’ the Government Equalities Office (GEO) has said, ‘use the power of transparency to help public bodies to fulfil the aims of the Equality Duty to eliminate discrimination, advance equality of opportunity and foster good relations between different groups. This means that public bodies will be judged by citizens on the basis of clear information about the equality results they achieve … Public authorities will have flexibility in deciding what information to publish, and will be held to account by the people they serve.’ (quoted by Instead Consultancy 2011) This approach embodies a series of assumptions that imbue numbers with an almost magical status and power. First, it is assumed that relevant and useful data will be made available (despite the selection being in the gift of the very authorities that ‘the public’ are expected to challenge). Second, this model of transparent data and active citizenship assumes that the citizenry has the time, resources, and expertise to access the data and then analyze it. Finally, the approach takes for granted that public bodies will automatically change their behavior if the data reveal poor ‘equality results’. Unfortunately, in the real world, none of these assumptions is true.
Statistics do not simply lie around waiting for interested citizens to pick them up and use them. Numbers are no more obvious, neutral, and factual than any other form of data. Statistics are socially constructed in exactly the same way that interview data and survey returns are constructed i.e. through a design process that includes, for example, decisions about which issues should (and should not) be researched, what kinds of question should be asked, how information is to be analyzed, and which findings should be shared publicly. Even given the very best intentions (and notwithstanding the opportunity for game-playing and ‘creative book-keeping’ of the sort already documented above) at every stage there is the possibility for decisions to be taken that obscure or misrepresent issues that could be vital to those concerned with social justice. In view of the limits of space, a single – but important – example will suffice. It concerns racial justice and the question of access to, and achievement in, UK higher education. It is a scandal that ethnic minority kids are more likely to go to university than poor white ones The Telegraph (Kirkup 2015) White British pupils least likely to go to university, says research The Guardian (Khomami 2015) White British pupils fall behind ethnic groups in race for university: All minorities now more likely to go into higher education Daily Mail (Doughty 2015) These headlines appeared in the British daily press in November 2015 when an economic think tank (the Institute for Fiscal Studies – IFS) publicized a review of government figures showing the proportion of young people going into the university from different ethnic groups (Crawford and Greaves 2015). First, as we might expect when applying a CRT perspective that is sensitive to the positioning of White people at the heart of contemporary politics, it is striking that the relatively low rate for White students is the angle highlighted by all news outlets regardless of their political positioning. Including, for example, the most left-wing (Guardian) and right-wing (Telegraph and Mail) parts of the mainstream British media. A second important aspect to this story, that may surprise some readers, is that there is nothing new in the fact that White students are less likely to enter British universities than their peers in most minoritized groups. This pattern was already known 18 years before these headlines: ‘relative to their share in the population … ethnic minorities overall are now better represented in HE than whites’ (Coffield and Vignoles 1997 original emphasis). From the perspective of this paper, focusing on the mis/uses of numbers in race analyses, perhaps the most important aspect of the IFS report, and the associated newspaper headlines, is that a focus on access statistics in isolation gives an extremely partial, indeed biased, view of race and Higher Education in Britain. Simply looking at who goes to university ignores long-standing and significant race inequities in the status of the universities attended and the level of final degree achievement.
Figure 1 shows the likelihood of attending an elite research-intensive university in the UK (the so-called ‘Russell Group’ of universities).2 White and minoritized students appear to have roughly similar chances of attending elite universities if all minoritized students are lumped together in a single ‘non-White’ group, usually referred to as BME in the UK (Black and Minority Ethnic). However, if the minoritized students are disaggregated into smaller and more meaningful groups, some important differences emerge. Figure 2 compares the proportion of White young people entering Russell Group universities against the rate for the most- and least-likely minority ethnic groups, Indian and Black Caribbean students respectively.3 White British students are almost five times more likely to gain access to elite
Figure 1 and Figure 2. Omitted
research-intensive universities than their peers of Black Caribbean background. This is a sizeable inequality of opportunity but is invisible in calculations that simply aggregate all minoritized students (such as Figure 1) or which look at access to all universities regardless of their standing (such as the national headlines quoted above). The inflammatory headlines that proclaimed the ‘scandal’ of White rates of access to university (above) draw attention away from a further facet of race inequity in the system i.e. differing levels of achievement between ethnic groups. Table 1 shows the proportions of students in each main ethnic group attaining the different classes of degree available at the end of their undergraduate studies; ranging from the very best result (a first class degree) through to a ‘third’ or ‘pass’ degree classification. White students are more likely to gain a ‘First’ than any other group (22.4%); Black students are the least likely to be awarded first class degrees (8.7% of Black students overall). This means that the odds of White undergraduates achieving the highest degree classification are around three times higher than their Black peers.4 This is a significant ethnic inequality but, perhaps because the beneficiaries are White, it goes entirely unremarked in the press furore about the overall access statistics (above). It is clear, therefore, that there is nothing obvious, neutral nor simple about education statistics and race. In this section, we have reviewed official data that describe differences in university access and achievement in relation to the ethnic origin of undergraduates in British universities. The government, an economic think tank, and the mainstream media all chose to highlight the apparent under-representation of White students (when looking at access across the entire system). This played into the ongoing high-profile political and media narrative that paints White people as race-victims in contemporary Britain (see Gillborn 2008, 2010b; Sveinsson 2009 for critical commentaries). But a very different picture emerges if the data are questioned in relation to a critical understanding of past race inequities in education. Such a perspective prompts us to explore differences in the status of institutions and the levels of achievement at the end of higher education. In both cases, White students appear to do rather well and, in terms of achievement, better than every other group. Indeed, there is perhaps scope for further headlines questioning what is happening in British higher education when the ethnic group that is least likely to go to university nevertheless enjoys the best chance of achieving the top grade. Were this a minoritized group there might be headlines about ‘scandals’ and shocks but, since the group in question is White, their high attainment fits with the basic expectations of a White supremacist media and polity and so the pattern goes entirely unremarked. Big Data: big trouble? The world’s capacity to store, broadcast and compute information is growing exponentially. The numbers involved have already passed well beyond the scales we are used to in our everyday lives. Counting across all forms of storage, from mobile phone memory to DVD, Blu-Ray and hard disks, we estimate that the world’s installed capacity to store information will reach around 2.5 zettabytes this year … If we stored all this data on DVDs and piled them up, the stack of discs would stretch one-and-a-half times the distance from the earth to the moon. What’s more, this figure is growing by over 50% year-on-year. (Yiu 2012, 10) ‘Big data’ is an increasingly popular phrase used to describe sets of numeric data that are, according to its advocates, simply too huge for traditional forms of human analysis. Big Data
Figure 3 omitted
has become big business. A recent google search for the phrase produced almost 300,000,000 hits5 and governments on both sides of the Atlantic are investing heavily in the technology and talking up its transformative powers: Big Data is a Big Deal … Today, the Obama Administration is announcing the ‘Big Data Research and Development Initiative.’ By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning. (WhiteHouse.gov 2012) It is estimated that the big data market will benefit the UK economy by £216 billion and create 58,000 new jobs before 2017 … Universities and Science Minister David Willetts said: ‘Big data is 1 of the 8 great technologies of the future and a priority for government. It has the potential to transform public and private sector organisations, drive research and development, increase productivity and innovation, and enable market-changing products and services.’ (Department for Business, Innovation & Skills 2014) Big Data advocates promote a hard sell about the fabulous powers of Big Data. They describe a world where new possibilities are revealed by an analysis entirely driven by machines and where, most significantly, theories and human reasoning are rendered obsolete because the ‘numbers speak for themselves’: the following extract is from an article in Wired magazine, entitled ‘The End of Theory’, which did much to popularize the idea: This is a world where massive amounts of data and applied mathematics replace every other tool that might be brought to bear. Out with every theory of human behavior, from linguistics to sociology. Forget taxonomy, ontology, and psychology. Who knows why people do what they do? The point is they do it, and we can track and measure it with unprecedented fidelity. With enough data, the numbers speak for themselves. (Anderson 2008) The argument that numbers can now ‘speak for themselves’ is a popular refrain in Big Data discussions. Speaking on BBC radio in 2013, for example, author Kenneth Cukier stated: ‘We have to let the data speak for itself. (…) When we trust the data – look at the data – it is a little bit less biased – in some respects, not in all respects – than we are. And therefore it can find correlations that we simply, as human beings, can’t because we have limited capacity (…) the vast amount of data has expanded, we now have to give it to the machine to do what it does best, and that is parse through it to come up with insights.’6 Cukier’s emphasis on correlations echoes part of Anderson’s argument from Wired: ‘Correlation is enough.’ We can stop looking for models. We can analyze the data without hypotheses about what it might show. (Anderson 2008) This is a deliberate and self-conscious rejection of the traditional warning that correlation should not be mistaken for causation. When Big Data advocates ask us to ‘trust the data’ they paint a picture of analysis as an almost mystical process that takes place inside machines and is too complex for human beings to comprehend: ‘We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot’ (Anderson 2008). As we noted at the very start of this paper, however, algorithms are not free from bias: ‘Even algorithms are biased against black men’ (Naughton 2016; see also Larson et al. 2016). And the reason that algorithms can be racist is that they are created and interpreted by human beings, many of whom share commonly held racist stereotypes. As we have argued above, all data is manufactured and all analysis is driven by human decisions. Although ‘Big Data’ advocates proclaim its insight and authority with almost evangelical fervor, the limits of the approach can be found lurking in the small print. For example, in a book whose subtitle proclaims Big Data as a ‘revolution that will transform how we live, work and think’, Cukier and his co-author accept (contrary to Anderson’s proclamation of the ‘end of theory’) that: ‘… big-data analysis is based on theories, we can’t escape them. They shape both our methods and our results. It begins with how we select the data. Our decisions may be driven by convenience: Is the data readily available? Or by economics: Can the data be captured cheaply? Our choices are influenced by theories. What we choose influences what we find … (Mayer-Schonberger and Cukier 2013, 72 emphasis added). This echoes our key argument that all data gathering and analysis is shaped by theories and beliefs that are susceptible to racial bias. In the next part of the paper, we set out some ideas for how the analysis of quantitative data might usefully be informed by the principles of Critical Race Theory (CRT)