Free beta access. 50 free AI tokens each week. Optional top-ups available. See what is included
Opening Word document…

Economic rationality is false – meta-analysis of existing studies proves that there is little evidence that demonstrates effective predictions

Nitsch et al 2022 - Comparative Psychology, Institute of Experimental Psychology, Heinrich-Heine-University Düsseldorf, Düsseldorf, Germany

Felix J. Nitsch, Luca M. Lüpken, Nils Lüschow, and Tobias Kalenscher July 26, 2022 “On the reliability of individual economic rationality measurements” Psychological And Cognitive Sciences 119 (31) e2202070119 https://doi.org/10.1073/pnas.2202070119

In the present paper, we investigated the reliability of behavioral measurements of rationality as a characteristic of individual decision makers. Across multiple original and published datasets, we found that the reliability of the two most prominent rationality indices (and variations thereof) is moderate to poor. This result held independent of the choice domain (social choice, food choice, choice under risk, or choice under ambiguity), choice complexity (two or three goods), study context (laboratory or online), incentivization (incentive compatible or hypothetical), study population, sample size, task structure, measurement length, and time gap between measurements. Hence, given data from multiple datasets with sufficient methodological diversity, our conclusions not only apply to a specific configuration of rationality measurements but speak with reasonable generality for contemporary research practice. More broadly, our results align with recent work on the reliability of measurements of risk preferences, sensitivity to losses, and self-regulation (33–35).#

Reliability indicates how much of the total variance in the variable of interest is attributable to true difference and not caused by measurement error. Hence, one potential explanation of moderate to poor reliability could be the presence of high measurement error (large denominator in the fraction) in the revealed preference methodology (e.g., 38). Our data offer two arguments against this explanation. First, allowing participants to revise a subset of choices (i.e., fixing potential mistakes, a source of measurement error; see study 3) did not increase reliability. Second, an analysis of the variance components in the data tentatively suggested that within-subject variance, as a proxy for measurement error, was sufficiently low for measurements with at least 20 trials.

Another explanation for moderate to poor reliability is a lack of true differences between participants: it could be possible that it is difficult to distinguish between participants because they do differ enough with respect to economic rationality. In line with this explanation, most participants across all datasets descriptively behaved with high consistency, and taking individual measurements of CCEI and HMI yielded approximately two times worse predictive accuracy for another measurement within the same individual than simply assuming the population mean (SI Appendix). In conjunction with the absence of high measurement error, this tentatively suggests that the low reliability of contemporary measurements of individual rationality (that is, the inability to distinguish between individuals) was indeed driven by a lack of interindividual differences in rationality.ǁ

As has been argued previously for other behavioral measurements, the lack of reliability poses a challenge to the contemporary search for sociodemographic or psychological correlates of economic rationality. Pragmatically speaking, our results show that a simple increase of trials or using a different task interface is not sufficient to fix this problem (unless the sample size is increased substantially); rather, individual differences must be increased. Possible avenues to explore here are, for instance, to ask participants to make decisions under stress or time pressure, increasing the difficulty of the decisions or using a manipulation (i.e., a between-groups design).

Sign in to recut, update, or save this card.