How not to combine ethnography and experiments in the study of moral judgment
In his latest Blog post, Hugo Mercier, discusses Clark Barrett et al.’s paper in PNAS: “Small-scale societies exhibit fundamental variation in the role of intentions in moral judgment.” (available here.)
Unlike Hugo, I don’t find this piece of work fascinating. In fact, given that excellent scholars I respect and admire have invested a good amount of effort in this work, I am quite disappointed, disappointed enough that I could write a long post detailing what I see as many serious theoretical and methodological weaknesses in this article, and too disappointed to bother to do so. Still, prodded by Hugo, I will react. To begin with, let me quote the way the authors describe what they see as the significance of their article:
It is widely considered a universal feature of human moral psychology that reasons for actions are taken into account in most moral judgments. However, most evidence for this moral intent hypothesis comes from large-scale industrialized societies. We used a standardized methodology to test the moral intent hypothesis across eight traditional small-scale societies (ranging from hunter-gatherer to pastoralist to horticulturalist) and two Western societies (one urban, one rural). The results show substantial variation in the degree to which an individual’s intentions influence moral judgments of his or her actions, with intentions in some cases playing no role at all. This dimension of cross-cultural variation in moral judgment may have important implications for understanding cultural disagreements over wrongdoing.
Sound promising but what does this study really show about moral judgment across culture, if anything?
The word “moral” appears more than an hundred times in the article, but the concept is not discussed at all: does any notion of “moral” have cross-cultural relevance and, if so, which notion? Not discussed. Which notion(s), if any, correspond(s) to “moral” in the societies compared? Not discussed.
Food taboos and disgusting food (in societies where there no food taboos) are considered on par, and both as unproblematically moral, two less-than-obvious decisions but no discussion.
So, to begin with, it is not that clear what the article is really about.
Participants were presented with vignettes describing some objectionable actions such as a theft and were asked in particular:
- In your opinion, how good or bad was what [the agent] did?
- When people discover what happened, what will people think of [the agent] — will they think he is a good person or a bad person?
- In your opinion did [the agent do this] on purpose, or by accident?
- In your opinion, do you think [Agent] should be rewarded or punished?
I very much doubt that the relevant notions ("bad action," "bad person," "on purpose," "reward" and "punishment") are identical across cultures or close enough that people’s response can be numerically compared. This is not discussed, nor is the way people in different cultures might have interpreted their task.
Still, Hugo is impressed by the case of the Yasawans (Fiji islanders), who, it seems, “judge equally harshly a series of moral wrongdoings irrespective of whether they were committed intentionally or not”. Well, as figure 3 shows, the Yasawa, quite unlike the other groups, tended to judge with the exact same mildness all actions presented, theft, food taboo violation, poisoning, bodily harm, whether they were done intentionally or not. Or maybe, more plausibly, they thought that answering in the same manner (by pointing to the middle of a horizontal five-points response scale) to all these questions was the right thing to do. How Yasawans and other participants might have understood the task and the proper way to perform it is not discussed.
Still, the authors rightly note:
Interestingly, Yasawa is a society in the Pacific culture area where mental opacity norms, which proscribe speculating about the reasons for others’ behavior in some contexts, have been reported.
Such kind of relevant ethnographic observation should have been provided in much greater detail to help interpret not just the Yawasans responses, but all the experimental evidence presented. I have little doubt that the result of putting things in a rich enough ethnographic perspective would have, on the one hand, allowed fine-grained qualitative comparison and made the work much more interesting and, on the other hand, would have radically put into question the numerical comparability of the cases.
Just as an anecdotal illustration, Joe Henrich and his collaborators have published several articles on Yasawa culture and society that, unlike this one, combine ethnography and experiments in a useful and sometimes ground-breaking way. In Rita Anne McNamara, Ara Norenzayan & Joseph Henrich (2016) Supernatural punishment, in-group biases, and material insecurity: experiments and ethnography from Yasawa, Fiji (Religion, Brain & Behavior, 6:1, 34-55) in particular, there is, in passing, the following observation:
Yasawans often see norm violations against increasingly distant outsiders as increasingly permissible (Henrich, 2008); although villagers are generally friendly and hospitable to everyone, they also find it more acceptable to steal from high-end tourist resorts than known members of the village. While the resorts regularly employ locals, many villagers are fired for stealing shortly after starting work (this may also be related to the traditional needs-based distribution and redistribution routinely employed among Yasawans, as documented in Gervais, 2013).
Shouldn’t such observations be brought to bear on the interpretation of Yasawans judgment on theft, especially since the theft described in the vignette is said to take place among people who do not know each other, hence, realistically, not among Yasawans who are all or nearly all acquainted with each other?
Instead of a useful combination of relevant ethnographic details and experiments, we get a study full of detailed statistics (without which, I reckon, it would not have been published in PNAS) and very little useful content. The authors, being serious scholars, are very prudent in drawing any theoretical conclusion from their work. The strongest conclusion they draw is:
Our findings do not suggest that intentions and other reasons for action are not important in moral judgment. Instead, what they suggest is that the roles that intentions and reasons for action play in moral judgment are not universal across cultures, but rather, variable. One way of interpreting this is that reasoning about the sources of an agent’s action—using theory of mind and other evolved abilities—is universally available as a resource for moral judgments, but it might not always be used in the same way, or even at all, in particular cases.
I would just qualify this commonsense conclusion by making the trivial point that a factor such as taking intentions into account in judging actions and actors can both be universal and have variable manifestations across cultures. So, what do we learn?