Free beta access. 5 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Lack of AI transparency undermines public trust

Putoni et al. 21 (Stefano, Behavioral scientist at the Rotterdam School of Management, Erasmus University, Director of the Psychology of AI Lab at the Erasmus Centre for Data Analytics, Rebecca Walker Reczek, Berry Chair of New Technologies in Marketing, Markus Giesler, Consumer researcher and Professor of Marketing at the Schulich School of Business, Simona Botti, PhD in philosophy, “Consumers and Artificial Intelligence: An Experiential Perspective”) Jan 2021 https://web-s-ebscohost-com.proxy.lib.umich.edu/ehost/detail/detail?vid=2&sid=fac31364-8601-4729-91f3-c78d18a8b3b0%40redis&bdata=JnNpdGU9ZWhvc3QtbGl2ZSZzY29wZT1zaXRl#AN=147580050&db=bth acd

The listening capability enables AI systems to collect data about consumers and the environment in which they live. We conceptualize the resulting experience as "data capture," which includes the different ways in which data are transferred to the AI. Data can be intentionally provided by consumers, albeit with different degrees of understanding of the process: consumers share data when there is little or no uncertainty about how the data will be used and by whom, or consumers surrender data when this uncertainty is high ([147]). Data can also be obtained by AI from the "shadows" consumers leave behind when they engage in daily activities, as in the case of a shopper perusing a store equipped with facial recognition technology or of an iRobot Roomba creating a map of a residential space ([89]). The data capture experience provides benefits to consumers because it can make them feel as if they are served by the AI: the provision of personal data allows consumers access to customized services, information, and entertainment, often for free. For example, consumers who install the Google Photos app let Google capture their memories but in return get an AI-powered assistant that suggests context-sensitive actions when viewing photos. Access to customized services also implies that consumers can enjoy the outcome of decisions made by digital assistants, which effectively match personal preferences with available options without having to endure the cognitive and affective fatigue that decision making can entail ([ ]). Finally, access to customized services offers unprecedented opportunities for self-improvement. Consider one of the projects within Alphabet, in which data from smartphones, genomes, wearables, and ambient sensors are combined to drive personalized health care ([86]).Despite AI's ability to predict and satisfy preferences, consumers can feel exploited in data capture experiences, mainly because they do not understand AI's operating criteria. This can be attributed to several features of AI. First, the modalities of data acquisition are becoming increasingly intrusive and difficult to avoid. Second, even when consumers intentionally share information, they are not aware of how this information is aggregated over time and across contexts. Finally, data brokers are largely unregulated and often lack transparency and accountability ([59]). As a result, data capture experiences may threaten consumers' ownership of personal data and challenge personal control, that is, the feeling that events are determined by the self rather than by others or by external forces and can be stirred toward desired outcomes ([36]). We examine the consequences of this loss of control next from both a sociological and psychological perspective. In popular culture, lack of ownership over personal data has been frequently associated with a loss of personal control stemming from technology's threatening potential to enable monitoring of human behavior. Stories such as George Orwell's 1984 or Philip K. Dick's Minority Report envision systems of oppression in which, due to lack of privacy and constant surveillance, people can no longer control their destiny. This dystopian imagination is echoed in sociological scholarship that associates data capture with the rise of a capitalist marketplace in which private information becomes the central form of capital ([157]).Such dystopian concerns strike a resonant chord when considering Google's move in the early 2000s to transform consumer data from a by-product into an economic asset that formed the basis of a new type of commerce driven by the ability to colonize the consumer's private experience. This commerce contributes to a surveillance marketplace, in which data surplus is "fed into advanced manufacturing processes known as 'machine intelligence' and fabricated into prediction products that anticipate what you will do now, soon, and later" ([157], p. 14, italics in the original). To illustrate the power of this commerce, targeted ads based on personality characteristics inferred from the analysis of Facebook likes in combination with online survey questions can increase conversion rates by about 50% ([102]). In 2018, Facebook's revenues from the sales of such tailored ads was close to $56 billion ([111]). From the perspective of this narrative, not only are technology companies continually required to find new ways to make monitoring and surveillance palatable to consumers by linking it to convenience, productivity, safety, or health and well-being ([10]), but they must also constantly push the boundaries of what private information consumers should share ([55]) through a complex landscape of notifications, reminders, and nudges intended to initiate behavioral change. Thus, as consumer behavior becomes increasingly retailored to the exigencies of behavioral futures, AI can transform consumers into subjects who are complicit in the commercial exploitation of their own private experience, thereby undermining personal control and promoting the concentration of knowledge and power in the hands of those who own their information. Data capture experiences are characterized by an underlying tension: consumers recognize that data capture allows AI to serve them through customization, but AI's inherent lack of transparency makes them feel exploited. These feelings of exploitation are fueled by actual and perceived loss of personal control, with important psychological consequences ([17]). The first of such consequences is negative affect, which can turn into demotivation and helplessness. Consider the case of Leila, a sex worker who shielded her identity on her Facebook account and reported being shocked to see some of her regular clients recommended by the "People You May Know" function. According to Leila, "the worst nightmare of sex workers is to have your real name out there, and Facebook connecting people like this is the harbinger of that nightmare." For Leila, like for domestic violence victims or political activists, privacy invasion is not only frightening, it may become a matter of life, death, or time in jail ([73]). As being in control is a basic need and a precondition of psychological welfare ([93]), the second consequence of loss of personal control may be moral outrage. Consider the case of a German consumer who requested his own data from Amazon and received transcripts of Alexa's interpretations of voice commands, even though he did not own any Alexa devices. The consumer relayed his story to a local magazine, which attempted to identify the consumer whose privacy had been compromised. The magazine staff involved in this experience described it as follows: "[we were able to] navigate around a complete stranger's private life without his knowledge, and the immoral, almost voyeuristic nature of what we were doing got our hair standing on end" ([24]). The third consequence of loss of personal control relevant to data capture experiences is psychological reactance, a state in which a person is motivated to restore control after a restriction ([21]), which causes more negative evaluations of and hostile behaviors toward the source of the restriction. In marketing, reactance can decrease the likelihood to repurchase and follow recommendations ([48]). Illustrating reactance in AI data capture experience is Danielle, a U.S. consumer who installed Echo devices throughout her home, believing Amazon's claims that they would not invade her privacy. When one of her Alexas recorded a private conversation and sent it to a random number in her address book, Danielle said "I felt invaded" and concluded, "I'm never plugging that device in again, because I can't trust it" ([76]). In summary, consumers may experience data capture as a form of exploitation: whereas technology companies, firms, and governmental agencies gain financial and political power, consumers lose ownership of their data and feel a loss of control over their lives. As we discuss next, managers should gain a better understanding of feelings of exploitation, as they prevent consumers from seeing the value firms can provide through data capture. This understanding starts at the organizational level and is then translated into decisions about experience design. A central programmatic task in addressing the issue of consumer exploitation in AI data capture experiences involves determining and enhancing the organization's level of awareness regarding the sociological and psychological costs raised in the previous sections. Companies should strive toward greater organizational sensitivity around consumer privacy and the current asymmetry in the level of control over personal data. For instance, they should use netnographic observation or sentiment analysis to listen empathetically and at scale to consumers who have experienced exploitation in AI data capture experiences. Furthermore, rather than accepting the surveillance society narrative at face value, firms can use these tools to understand when, how, and whether their own data capture experiences play into versus subvert this narrative. Likewise, companies should draw on insights by privacy scholars and activist movements to question their taken-for-granted beliefs. In doing so, for instance, companies could realize that their own view on privacy default settings might differ markedly from that of a vulnerable consumer group and adjust their processes accordingly ([101]).

Sign in to recut, update, or save this card.