White hyperreality thesis is wrong. Race is multimodal, not just semiotic.

Hall’s structuralist orientation to race, while groundbreaking and profoundly important, reinforced some of the biases and preferences of the Western thought. First, in arguing that racial meanings are coded into visible physical differences, Hall’s conceptualization of race emphasized the predominance of vision in the construction of racial differences. That is, seeing is believing—we see different skin color, hair texture, and body shape to confirm some underlining, presumably biological and innate, differences. Then again, as expressed in colorblind racism—not seeing is believing—dismissing the cultural practices of race and realities of racism extends and carries on inequalities, prejudice, discrimination, and violence (Doane and Bonilla-Silva, 2003). While it is still commonly assumed that racial differences are visually self-evident, various scholarship has addressed that the construction of racial differences involves more than vision and the role of other intersecting senses cannot be underestimated (Boswell, 2017; Fretwell, 2020; Obasogie, 2014; Sekimoto and Brown, 2020; Smith, 2006). Second, Hall’s approach to race as a language resulted in treating the human body as text to be inscribed, read, and interpreted. The idea is that racial difference is an “inscription” on the body, or a text written on the surface of the body, which can be “read” and “decoded.” Yet, if we turn our eyes—and other sense organs—to the lived experiences of racialized subjects, we realize race is lived as more than a surficial inscription on one’s body. The impact of racialization is more than symbolic or ideological, but corporeal and visceral (Ahmed, 2007; Fanon, 1967; Lee, 2014; Menakem, 2017; Samatar, 2015; Yancy, 2012). Race inhabits the racialized body and mind as something more than a language that can be systematically interpreted (and potentially re-written), but as something more material and concrete, physically installed and habituated (Bourdieu, 1977; Ngo, 2017; Saldanha, 2007). In his lecture, Hall (1997) said:

The body is a text. And we are all readers of it. And we go around, looking at this text, inspecting it like literary critics…We are readers of race, that what we are doing, we are readers of social difference. And the body hair, which you know is sighted as if, this is what terminates the argument. When you say race is a signifier. No it is not! See the folks out there they are different! You can tell they are different. Well, that very obviousness, the very obviousness of the visibility of race is what persuades me that it functions because it is signifying something; it is a text, which we can read (15)

While Hall used the metaphor of body-as-text as a way of critiquing the primacy of vision that fools our perception of visible differences as biologically meaningful (and thus justifies its political application), his theorizing did not go far enough to interrogate the role of the living, feeling body that always does more than reading a metaphorical racial text. “Reading” connotes a linear activity rather than a dynamic, fluid, and mutual interaction; “text” inevitably connotes a static object to be decoded, rather than an emergent phenomenon that materializes in time and space. While the metaphor of body-as-text speaks some truth to the politics of racialization, it inadvertently fixes the analytic lens intended for understanding the “floating” nature of race as a signifier. The metaphor of text, reading, and language falls short of grasping the materiality and movement of lived human bodies that participate in the process of meaning making about race.

Finally, the body-as-text metaphor also emphasizes the act of reading as a mode of comprehension and meaning making, and thus privileges cognition, mind, and thinking while subordinating emotion, body, and feeling (De Souza Santos, 2018). Reading, which relies predominantly on vision, is possible because of the distance between the subject (a reader) and the object (text); it requires the body to be still and stationed; and it presupposes some kind of rationality. Reading connotes a linear cognitive activity that takes place in the internal space of one’s mind. Reading, as a metaphor of how we make sense of race, overlooks the fact that racial meaning is communicated beyond the visible signs and conscious cognitive interpretation. What makes race so hegemonic and enduring as a social construct is its impact on the more subconscious, yet felt and material, process of meaning making that is not as intelligible as reading a text. In everyday experiences, we produce racial differences through active perception, participation, and sense-making. We co-create and co-participate in the act of racialization as mobile, feeling, and sentient subjects. Hall’s metaphor does not capture the nuances of such embodied, more-than-representational participation in the world of race. Furthermore, the race-as-language or race-as-text model perpetuates the binary between subject (the one who reads race as text) and object (the racialized body that is read as text) in addressing issues of race. This subject-object dichotomy makes it extremely difficult to centralize and humanize the subjectivity of racialized Other because, by definition, to be racialized means to assume the position of an object/text. Such approach evades a deeply existential and phenomenological question of how race and racism infiltrate one’s sense of self.

Race as multimodal assemblage

Multimodality, as a theoretical concept and pedagogical tool that attends to the multiple ways in which meaning comes together and is held together, allows scholars to examine race as more than a language or text, but as a dynamic and hegemonic assemblage of visible and invisible communicative action and resources. Multimodality, at its core, is an epistemological orientation that goes beyond purely textual, visual, and representational modes of knowing. It encompasses embodied, emplaced, dynamic, and felt dimensions of social interaction and communication. The theoretical exigency that drove the development of multimodal scholarship in the past several decades coincides with the axiological urgency to address the lived experiences of racialized subjects as they are lived—not just read as text. That is, the work of “fixing” race is done multimodally, just like other forms of meaning making and communicative practices.

The turn to multimodality in addressing race joins the ongoing struggle to bring the otherwise unintelligible voices and sensations of the marginalized Other into the scope of analysis and knowledge production. As co-editors of this special issue, we came to the term multimodality with an intellectual impulse to go beyond the representational way of knowing (Sekimoto, 2012; Sekimoto and Brown, 2020). As racial minorities and speakers of American English as a second language/dialect in the United States, our senses of otherness (or alterity) materialize in the space where conventional linguistic expressions fail to capture our lived experiences and sensations. We took a detour to the world of senses – our senses seem to be making meaning of everything we perceive, in ways that are not brought to our cognitive awareness or linguistic articulation. Multimodality, as a philosophical orientation, calls us to attend not only to things that are “more than words,” but rather to extend our imagination to conditions and experiences that have no (intelligible or legitimate) words, labels, or categories to describe. In fact, the depth of racialization and the impact of racism reside in this unnamed and unintelligible space.

Multimodality allows for a different conceptualization of race – it is not a text written with the arbitrary and slippery language of race, but an assemblage of multiple modes that constitutes a racial world that is not only “read” but also “lived” and “felt” by those who live in it. It is not only a text, but an entirety of the environment—an ecosystem of differences—that is built according to the narratives, benefits, and logics of the powerful. In response to some of the limitations of the race-as-language model, multimodality challenges the privileging of vision as a primary mode of knowledge production about race. Multimodality allows for a re-conceptualization of the “body-as-text” into the body-as-sentient, body-as-kinesthetic, body-as-emplaced, and body-as-interactive. We no longer need to rely on a metaphor to describe the body, because the body is a tangible, material, and breathing presence that actively participates in communication. Finally, multimodality goes beyond “reading” as a primary metaphor of racialized interactions by allowing for other modes of engagement as part of constitution of meaning. Communicative modes such as space, body, gaze, gestures, colors, images, sounds, and fonts necessitate mobilizing the verbs like moving, walking, looking, listening, touching, and feeling, beyond the act of reading in stillness.

Racializing multimodality: A methodological consideration

The turn to multimodality as a kaleidoscopic and dynamic lens through which we examine race in communication poses a set of methodological questions. As Kress, et al. (2001) asks, “what are the affordances of each mode”? (1, original emphasis) and how does each mode combine with other modes to construct a social world that continues to grant race its legitimacy and power? How is what we conceive of as a “mode” and its affordance racially charged or associated with racialized meanings? For example, gaze is a mode of communication that is often used to analyze the gendered and racialized relations of power—because the gaze constructs the relations of power between the one that looks (subject), and the one that is looked at (object). A researcher would lose a lot of insight if they only focused on what is verbalized in words and not what is communicated by the act of looking. Racialized subjects both speak and are spoken through the gaze—there is a different kind of affordance in gaze, in comparison with, for instance, spoken words or written text. Spoken words or written text, however, have been traditionally more prioritized as a site of meaning making as opposed to more silent, subtle, and embodied forms of communication. That is, what counts as a “mode of communication” worthy of analysis and comprehension is a political question that may render certain racialized experiences unacknowledged and unintelligible. Furthermore, gaze’s communicative affordance is experienced quite differently by those who are marginalized in society. The act of looking (back) may give them a space of resistance against the dominant gaze, but also it may risk their safety and incite retaliation. Simply put, not all modes provide the same kind of affordance for all people.

The attention to race through multimodality presents an important shift from “the what to the how of perceiving, which is where ‘ways of sensing’ or ‘techniques of perception’ comes in” (Howes, 2022: p. 169, original emphasis). That is, analyzing race must come with a careful consideration of not only the material object produced by symbolic practice (words, speech, images, sounds, etc.) but also the perspectives, values, and ideologies that inform how symbolic practice is engaged and interpreted. The recent publication of White Sight: Visual Politics and Practices of Whiteness by Nicholas Mirzoeff (2023) (see the book review by Sobande in this special issue) shows the intricate role of visual culture and ways of seeing in the historical construction of white supremacy. That is, perceiving race is not merely a matter of signifiers (skin color, hair, etc.) and signified (superior, inferior, etc.), but more fundamentally about the situated perspective that produces what Mirzoeff calls “white sight.” If every mode presents a certain affordance, such affordance is not a neutral or open-ended space of possibility in which any meaning production can be explored; rather, the affordance of a mode is shaped by a situated perspective of people who deploy it.

The affordance of gaze, for example, has been deployed by the Western subjects to establish a perspective that divides the world into the self (the West) and the other (the Rest) (Said, 1979). In multimodal theorizing of race, we must attend to how a communicative mode takes on a racialized filter, or how a mode works as a racializing filter. We encourage our readers to attend to how the way multimodality is conceptualized in various contexts may racialize certain bodies, knowledge, and practices—as a mechanism that establishes normative ways of communicating that implicitly or directly supports the hegemony of whiteness. In considering the methodological implications of integrating multimodality in addressing race and racism, we ask what counts as “modes,” what constitutes a “meaning,” as well as whose perspective is assumed in the act of signification.

Overview of the contributions

This special issue recognizes the attempts made by scholars and practitioners to expand the ways in which we understand the nature of race and racism beyond the metaphors of text, language, and reading. The contributions draw the reader’s attention to the modes of meaning making about race that may conventionally be ignored or overlooked. If “communication is inevitably multimodal,” communicating about race is also a multimodal accomplishment (Kress, et al., 2001: 3). It is important to examine the affordances of each mode used in the social construction of race, while understanding how each mode interacts with, and traverses across, multiple other modes. Developing a multimodal understanding of race is crucial for any anti-racist efforts—our attention cannot be isolated to any single battle against certain kind of language, images, law, or representation. The contributions in this special issue reveal how race exists in the multiplicity of modes.