Meta-analyses comprehensively refute the impact.

A substantial proportion of replications in recent large-scale efforts have failed to support the original finding (Nosek et al., 2022). Less than 40% of 100 replications from three top psychology journals were considered successful across a variety of criteria (Open Science Collaboration, 2015). Likewise, 13 of 21 social science experiments published in the journals Nature or Science were interpreted as replicating successfully based on observing statistical evidence (p < .05) in the same direction as the original study (Camerer et al., 2018). Using similar criteria, the “Many Labs” projects conducted high-powered replications of non-randomly sampled studies, yielding variable rates of success: Many Labs 1 (Klein et al., 2014): 10/13, Many Labs 2 (Klein et al., 2018): 14/28, Many Labs 3 (Ebersole et al., 2016): 3/10, Many Labs 5 (Ebersole et al., 2020): 2/10. There are many possible contributors to failures to replicate, including publication bias or p-hacking in original research, unidentified moderators that differ between the original and replication, and failures to effectively implement the replication studies. Shortcomings in implementation could occur by failing to transfer knowledge of key features of the study methods to replication teams.

We sought to evaluate whether involving original authors in selection and design could improve replication success. To do so, we selected an important theory that is also considered sensitive to the procedures necessary to elicit its effects: Terror Management Theory (TMT; Greenberg et al., 1986, 1994). TMT considers the psychological consequences of being reminded of one’s impending death. This topic has spawned hundreds of publications, some with more than 1,000 citations. TMT states that as humans evolved self-awareness, they also came to know that their death is inevitable. To avoid preoccupation with thoughts of death or feelings of meaninglessness, one must manage the potential terror caused by this knowledge (Becker, 1973, 1975). Greenberg, Pyszczynski, and Solomon (1986) proposed that self-esteem is the buffer to these intrusive thoughts, and that the purpose of self-esteem is to manage terror related to mortality. TMT has been applied to understand human activities such as religion (e.g., belief in an afterlife grants literal immortality, alleviating mortality terror; Jonas & Fischer, 2006), cultural identity (e.g., belief in being part of a greater good that will persist after death; Greenberg et al., 1990), and inter-group conflict (Burke et al., 2010). TMT experts also indicated that there was substantial nuance required in implementing a successful TMT study, and at least one doubted it could be captured in a “Many Labs” style project at all. These nuances include how the experimenter delivers the experimental script (tone, manner), precise design of materials, accounting for contextual details, and other aspects which were refined over years. However, there has never been a systematic investigation of the replicability of TMT findings.

To investigate whether author involvement could improve replicability, we engaged experts in three phases of research design. First, we conducted a community-based search for an important area of research that requires expertise for effective implementation and had experts willing to contribute to replication. TMT met these criteria.

Second, we consulted with TMT experts – Tom Pyszczynski and Sheldon Solomon – to identify a seminal study appropriate for replication (Jeff Greenberg later contributed to the design of the study materials). Candidate studies had to take 30 minutes or less, be administered on pencil-and-paper or computerized, and preferably have two or fewer between-subjects conditions to maximize statistical power. We sought studies that had room for researcher flexibility in design, were theoretically central to TMT, and had reasonably high expectations of replicability. In consultation with experts, we selected Study 1 from Greenberg, Pyszczynski, Solomon, Simon, and Breus (1994). This paper is highly influential (1252 citations on Google Scholar as of January 14, 2022), and Study 1 used a prototypical mortality salience manipulation: writing about one’s own death, which 79.8% of studies in the field of TMT have used (Burke et al., 2010).

Third, original authors were instrumental for designing the “Author Advised” version of the replication protocol. The goal was to leverage all possible expertise to design a study with the greatest chance of replicating the original effect. All parts of the description, implementation, and analysis plan for this Author Advised protocol were reviewed by at least one original author before data collection began.

<<TEXT CONDENSED, NONE OMITTED>>

To investigate whether author involvement increases replicability, we recruited 21 teams and randomly assigned them to either administer the Author Advised protocol or to develop and administer their own “In House” protocol to samples that they recruited. In House protocols were designed without contact with the original authors, other teams participating in the project, or other domain experts. Each lab was responsible for collecting data from at least 80 participants to ensure a high degree of statistical power when aggregating across sites.1 At some labs, when separate principal investigators were identified to administer independent replications, both In House and Author Advised protocols were conducted. With this design, we examined the following questions: 1. Can we successfully replicate a central finding supporting TMT? We replicated Study 1 of Greenberg et al. (1994) in 17 labs with 1,578 participants (after exclusions). At minimum, this provides a high-powered test, for all but very small effect sizes, of this important finding. 2. Does original author involvement improve replicability? The selected study (Greenberg et al., 1994, Study 1) left ample room for non-experts to miss potentially critical aspects of implementation. For example, the original paper deviated somewhat in materials from the eventual Author Advised replication implementation and contained little of the advice regarding lab context and setup. We tested whether the Author Advised protocol was more effective at eliciting statistically significant and larger effect magnitudes than In House protocols. 3. Does a standardized, author-reviewed protocol produce less variability across data collection sites? We hypothesized that a standardized implementation of the Author Advised protocol across sites would result in less variation in results compared to the In House protocols because the In House protocols would likely have substantial variability in procedures. Method Study 1 of Greenberg et al. (1994) provided evidence that reminders of death induce worldview defense, and that this effect was stronger when the reminders of death were comparatively subtle (i.e., thinking about one’s own death) than when reminders of death were more emotionally salient (i.e., thinking about one’s own death and reflecting on their deepest feelings about dying). In the original study, a total of 59 introductory psychology students in the United States were randomly assigned between five conditions. Our replication focused on the two conditions that were most effective in the original paper: the condition labeled “subtle own death salient” and the “TV salient” (control) condition. The outcome of interest was evaluations of pro- vs. anti-American essay authors. Specifically, participants in the death salient condition reported a greater preference for the pro-American essay author over the anti-American essay author, compared to participants in the control condition. Participants in both conditions were told that the study session was composed of two separate studies. In the “first study,” participants completed two filler measures and the mortality salience or control manipulation. Participants in the condition described as “subtle own death salient” wrote about the emotions they experienced when thinking about their own death, and about what would happen to their physical body as they were dying and once they were dead. Participants in the “TV salient” condition received similar writing prompts, but instead described the emotions they would experience while watching television, and what they thought happened to their physical body as they watched television. Participants then completed the Positive and Negative Affect Schedule (PANAS; Watson et al., 1988). In a so-called “unrelated second study” during the same session, participants read two brief essays ostensibly written by international students. One essay was pro-American and the other was anti-American. For the dependent variables, participants answered three questions evaluating each essay’s author, and two questions evaluating the essay itself. The present replication focused on evaluations of the author because that showed the strongest effect in the original study. A composite rating of the author was created by subtracting the mean of the three anti-American items from the mean of the three pro-American items. A pairwise comparison revealed that participants in the “subtle own death salient” condition showed a greater preference for the pro-American author than for the anti-American author (Mean Diff = 12.25). This was compared to participants in the “TV salient” (control) condition (Mean Diff = 1.64), t(53) = 4.87, p < .001, Cohen’s d = 1.34 (Greenberg et al., 1994). Notable Differences from Original. The replication included just two of the five conditions from the original (“subtle own death salient” and “TV salient”) and focused on just the evaluations of the essay authors (disregarding any evaluations of the essays themselves). Because this mortality salience condition may not be considered particularly subtle, and is described in the original paper as their usual mortality salience induction, we will refer to this condition simply as the mortality salience condition from here on. The replication studies were administered in the US Fall of 2016 and Spring of 2017. Procedure for Author Advised Protocol. All materials and instructions used in the study are available on the OSF page (https://osf.io/bq4n4/), as well as instructions and detailed procedures and advice provided to Author Advised sites. For the replication, we made several changes to the original protocol as requested by the original authors: 1. The Anti-American essay was adjusted to be more forceful and extreme, to ensure it conflicted with the average participant’s worldview. 2. The filler-task portion of the study was extended to include additional surveys and increase the delay between the mortality salience induction and essay evaluation. 3. The procedure tried to put participants into a relaxed, experiential mood. Precise language was included in the expert instructions packets and included recommendations such as selectively choosing relaxed research assistants to administer the study, using a covered box for handing in packets to ensure a feeling of confidentiality, separating participants into different cubicles or rooms, and having experimenters act and dress casually. 4. Additional demographic items were added to the end of the survey to apply exclusion criteria. The Author Advised protocol was always administered in person and in individual cubicles or small groups. When participants first entered the room, they were read a prepared script that included a cover story that the study consisted of two parts, and statements assuring participants of the anonymity of their responses to reduce demand characteristics. Participants were instructed to complete the first half of the survey, and then return the completed form to a covered box before being given the second survey. Then, participants were seated and given the second packet of materials. The first materials packet started by reassuring participants that their responses were confidential, and that they should respond naturally: “On the following pages you will find a series of personality, attitude and judgment questionnaires. There are no right or wrong, or good or bad answers; rather different responses reflect different personalities, attitudes and judgment styles. Please respond honestly and naturally to each question and complete the questionnaires in the order that they appear in the packet. Your responses to these questions are completely anonymous and will be used for research purposes only.” Then, participants completed a 23-item “personality inventory” included as a distractor, in which participants answered “Yes” or “No” to items such as, “Does your mood often go up or down?” They also completed a 12-item measure modeled after the Personal Need for Structure Scale (Thompson et al., 2001), which was also included primarily as a distractor. Participants responded on a 6-point scale (1 – strongly disagree to 6 – strongly agree) to items like “It upsets me to go into a situation without knowing what I can expect from it.” After this, participants completed the mortality salience or TV (control) induction. Prior to the induction text, participants were again reminded to respond naturally: “On the following page there are a couple of open-ended questions. Please respond to them with your first, natural response. We are just looking for people’s gut-level reactions to these questions.” Participants in the mortality salience condition then responded to two open-ended items disguised as “The Projective Life Attitudes Assessment.” The first item asked participants to “Please briefly describe the emotions that the thought of your own death arouses in you,” while the second item asked participants to “Jot down, as specifically as you can, what you think will happen to you physically as you die and once you are physically dead.” Participants in the TV (control) condition responded to nearly identical items, which instead asked about their emotions experienced while watching television and what happens to them as they watch television. Participants then completed the PANAS-X (Watson & Clark, 1994), a 60-item measure of current emotional state in which they indicated the degree to which they currently felt each of 60 emotions on a 5-point scale (1 – very slightly or not at all to 5 – extremely), and the Morningness-Eveningness Questionnaire (Horne & Ostberg, 1975) which is a 19-item scale assessing the degree to which participants performed better in the morning or evening. Example items included: “What time would you get up if you were entirely free to plan your day?” and, “What time would you go to bed if you were entirely free to plan your evening?” The purpose of these two scales was to provide “filler” time between the induction and the essay evaluations. This concluded the “first half” of the study and the first materials packets. After the participant handed in the first packet, they were provided the second packet of materials, matched by a participant number. Participants read a cover story that indicated the university had collected writing samples of impressions of America from international students attending the university. Participants then read two essays, in counterbalanced order across participants. One essay was relatively pro-U.S. and the other was relatively critical of the U.S. Participants rated each essay and its author on a five-item, 9-point scale (1 = “not at all” to 9 = “extremely). Example items include”To what extent do you think the essay makes valid points?” and “How intelligent do you think the person who wrote this essay is?”. Procedures for In House Protocols. The In House protocols were created independently by each participating lab. Instructions provided to these labs are available on the OSF page (https://osf.io/drfg2/). To the best of their ability, and using only the original paper, relevant literature, and any other publicly available resources, each lab designed their own replication protocol of the assigned study. The labs were prohibited from contacting the original authors, other experts, and the other participating labs. The only outside review was by the project leaders to confirm that the correct study was being replicated and that the resulting data collection would yield all variables necessary for the basic analysis plan. The In House protocols differed substantially, both from one another and as compared to the standardized Author Advised protocol. For example, 7 In House labs used the PANAS as a filler task between the manipulation and the dependent variables; the other 5 labs included no filler task at all. None of the In House labs used the same set of filler tasks as the Author Advised version, which included the PANAS and second unrelated questionnaire. All 12 In House labs collected data using a computer instead of pencil-and-paper, a design explicitly discouraged by the original authors for the Author Advised version. We summarize major attributes of the designs of all In House sites in Table 2. All materials used at In House sites and videos documenting their implementation are available at http://osf.io/8ccnw/ to facilitate review of the varying procedural implementations. Study Administration. Within each lab, participants were randomly assigned to either the mortality salience or control condition of the given protocol. At three universities, both protocols were administered by independent groups of researchers (to maintain blinding to the Author Advised materials). At those sites, the groups recruited participants separately, but randomized participants between conditions in their given protocol. Each collaborating lab was responsible for recruiting at least 80 participants for their assigned protocol. Participants were prohibited from participating in the study more than once. Provisions for Quality Control As in previous iterations of the “Many Labs” projects (e.g., Ebersole et al., 2016, 2020; Klein et al., 2014, 2018), we employed quality control and accountability standards for each lab’s data collection. Each site was required to film a mock session of the data collection in which the researcher filmed a mock participant walking through the steps as if they were actually taking the study. These videos allowed a more thorough documentation of the lab spaces and any idiosyncratic differences in protocol. In addition, sites were responsible for ensuring all experimenters in direct contact with participants were blind to the participant’s condition assignment. Collaborators also completed an experimenter survey assessing their expertise, expectations, and motivations for joining the collaboration. The survey and videos are available at https://osf.io/8ccnw/. At the end of the Author Advised protocol, participants filled out demographic information. Three of these items were included at the request of original authors: importance of American identity, country of birth, and race. In House protocols were asked to collect similar demographic information (see https://osf.io/drfg2/), but these were described minimally to avoid influencing design decisions. The American identity item was omitted entirely. As a result, demographic data from In House sites were not entirely comparable to demographic data from Author Advised sites. Participants Twenty-one labs from 18 universities participated in the project and collected data from a total of 2,281 participants. Several overall exclusions were made, each as specified by our preregistration: https://osf.io/4xx6w. First, data from four labs were excluded for collecting fewer than 60 participants.2 This excluded 158 participants. Next, some In House sites began data collection before the analysis plan was pre-registered due to their own deadlines. Data collected before the pre-registration were excluded from the confirmatory tests reported in this manuscript. This excluded a further 545 participants.3 From the remaining 17 labs and 1,578 participants, in adherence to our pre-registration, we excluded from all analyses participants who either failed to complete all 6 ratings of the essay authors, or who failed to complete both writing prompts within the mortality salience or control conditions (e.g., the between-subjects manipulation). The latter exclusion criteria applied only to participants from Author Advised sites, because the necessary data was not always available for In House sites. Thus, the usable sample included 1,550 participants (see Table 1 for a breakdown by lab). 841 participants (54.26%) were women and 442 participants (28.52%) were men; the remaining participants did not respond to the item, were asked about gender in a non-standard way, or chose a different response. The mean age was 19.91 years (SD = 2.49). Participant reported race was 946 (61.03%) White, 252 (16.26%) Asian, 112 (7.23%) Black or African American, 4 (0.26%) American Indian or Alaska Native, 7 (0.45%) Native Hawaiian or Pacific Islander, and 124 (8%) another category. The remaining participants did not report their race or their responses could not be recoded to match these categories. <<TABLE 1 OMITTED>> <<TABLE 2 OMITTED>> Analysis Plan The primary finding of interest from Greenberg et al. (1994) was that participants who underwent the mortality salience treatment showed greater preference for the pro-US essay author over the anti-US essay author as compared to the control condition. To assess whether the replication results support the original, within each lab we followed a similar analysis plan as in the original article, as specified by our pre-registration (https://osf.io/4xx6w). Scores from the three items evaluating the authors of the anti-American essays were averaged (α = 0.89) and then subtracted from the average of the three items evaluating authors of the pro-American essays (α = 0.89). An independent-samples t-test was then conducted comparing those in the mortality salient (MS) condition with scores from the “TV salient” (control) condition. We then analyzed these individual results meta-analytically to get an aggregate effect size across all labs. Supplemental exploratory (non-preregistered) analyses treating these as two separate dependent variables are available in the online supplement (https://osf.io/xtg4u/), and those outcomes do not qualify the conclusions offered here. In addition to the overall exclusions detailed previously in the “Participants” section, original authors requested some additional exclusions based on participant responses. Original authors were not entirely in agreement about what exclusions should be implemented. Therefore, as indicated in the pre-registration, we repeated our analyses under three different exclusion criteria: A minimal set of exclusions (Exclusion Set 1), and two sets of more strict exclusions recommended by original authors (Exclusion Sets 2 and 3). Because Exclusion Sets 2 and 3 were specifically recommended by original authors and could be considered part of their expertise, we did not inform the In House labs about these exclusions prior to data collection. This masking was to avoid influencing their decisions about data collection procedures, which had to be made independently of any outside help. However, as a result, none of the In House labs collected the data required to make these more strict exclusions because the items required were not present in the original target article. Therefore, analyses were repeated for each of the three exclusion sets for Author Advised participants. However, Exclusion Set 1 was used consistently across all analyses for In House participants. The exclusion sets consisted of: Exclusion Set 1: Exclude participants who did not complete both writing prompts and all six items evaluating the essay authors. This yields 1,550 participants (n = 699 Author Advised, n = 851 In House). This aggregate sample size gives us 95% power to detect a mortality salience condition effect of d = 0.18 in an independent samples t-test.4 <<TABLE 3 OMITTED>> Exclusion Set 2: All prior exclusions, and further exclude participants who did not identify as White or who indicated they were born outside the United States. This reduces the N to 1,229 participants (n = 378 Author Advised, n = 851 In House). This aggregate sample size gives us 95% power to detect a mortality salience condition effect of d = 0.21. Exclusion Set 3: All prior exclusions, and further exclude participants who responded lower than 7 on the American Identity item (“How important to you is your identity as an American?” 1 - not at all important; 9 - extremely important). This further reduces the usable N to 1,076 participants (n = 225 Author Advised, n = 851 In House). This aggregate sample size gives us 95% power to detect a mortality salience condition effect of d = 0.22. All data handling, exclusions, and computation of results within sites followed our pre-registered (prior to data collection) analysis plan on the OSF (https://osf.io/4xx6w). All results are reported with two-tailed tests of significance.5 Results Deviations from Pre-registered Analytic Plan Our pre-registered analytic plan specifies the use of a three-level meta-analysis, conducted in the MetaSEM R package (Cheung, 2014), to control for the clustering of effect sizes when independent teams ran both In House and Author Advised versions of the protocol at the same university. However, during data analysis we discovered that these models failed to optimize to a solution (OpenMX status1 = 5) which means that their output cannot be interpreted (see https://openmx.ssri.psu.edu/wiki/errors). This is likely because we did not have enough data within each cluster, as almost all sites ended up conducting only one study (e.g., administering only the Author Advised or In House protocol, not both). As such, we had to drop the clustering variable. The results reported below are thus a more common univariate meta-analysis, which is the model that most closely mirrors our originally planned analysis.

<<PARAGRAPH BREAKS CONTINUE>>

Research Question 1: Meta-analytic Results across All Labs (Random Effects Meta-analysis).

The most basic question is whether we observed the predicted effect of mortality salience on preference for pro- vs anti- American essay authors. To assess this, we conducted a random-effects meta-analysis. This analysis produces the grand mean effect size across all sites and versions. Regardless of which exclusion criteria were used, we did not observe the predicted effect, and the confidence interval was quite narrow: Exclusion Set 1: Hedges’ g = 0.07, 95% CI = [-0.03, 0.17], SE = 0.05, Z = 1.32, p = .187. Exclusion Set 2: Hedges’ g = 0.09, 95% CI = [-0.03, 0.21], SE = 0.06, Z = 1.49, p = .135. Exclusion Set 3: Hedges’ g = 0.09, 95% CI = [-0.04, 0.22], SE = 0.07, Z = 1.30, p = .193. Forest plots showing the effects for individual sites and the aggregate are available in Figures 1, 2, and 3 for Exclusion Sets 1, 2, and 3, respectively. These results indicate that, in the aggregate, we failed to replicate the predicted mortality salience effect.

<<FIGURE 1 OMITTED>>

There may have been a mortality salience effect at some sites and not others, so we next examined how much variation was observed among effect sizes (e.g., heterogeneity). This sort of variation did not exceed variation expected by chance (e.g., sampling variance) regardless of exclusion rule: Exclusion Set 1: Q(16) = 19.35, p = .251; Exclusion Set 2: Q(16) = 21.72, p = .152; Exclusion Set 3: Q(16) = 18.57, p = .292. This result suggests that any observed differences in effect size between sites are likely due to random noise, as opposed to true differences in effect size.

In sum, we observed little evidence for an overall effect of mortality salience in these replications. Additionally, overall results suggest that there was little or no heterogeneity in effect sizes across sites. This lack of variation suggests that it is unlikely we will observe an effect of Author Advised versus In House protocols or other moderators such as differences in samples or TMT knowledge. Even so, the plausible moderation by Author Advised/In House protocol is examined in the following section.

Research Question 2: Moderation by Author Advised/In House protocol

A covariate of protocol type (In House vs Author Advised) was added to the random effects model to create a mixed-effects meta-analysis. This is our primary model of interest, and the model most similar to the three-level mixed-effects meta-analysis that we pre-registered as our primary outcome.

This analysis again produces an overall grand mean effect size, and those were again near zero and relatively precisely estimated across all three Exclusion Sets: Exclusion Set 1: Hedges’ g = 0.07, 95% CI = [-0.03, 0.17], SE = 0.05, Z = 1.33, p = .182. Exclusion Set 2: Hedges’ g = 0.11, 95% CI = [-0.02, 0.24], SE = 0.07, Z = 1.71, p = .087. Exclusion Set 3: Hedges’ g = 0.13, 95% CI = [-0.03, 0.29], SE = 0.08, Z = 1.57, p = .117.