12 March 2013

A General Rule For Social Science Research

When he began to run the game it became immediately clear that Machiguengan behavior was dramatically different from that of the average North American. To begin with, the offers from the first player were much lower. In addition, when on the receiving end of the game, the Machiguenga rarely refused even the lowest possible amount. “It just seemed ridiculous to the Machiguenga that you would reject an offer of free money,” says Henrich. “They just didn’t understand why anyone would sacrifice money to punish someone who had the good luck of getting to play the other role in the game.”
The potential implications of the unexpected results were quickly apparent to Henrich. He knew that a vast amount of scholarly literature in the social sciences—particularly in economics and psychology—relied on the ultimatum game and similar experiments. At the heart of most of that research was the implicit assumption that the results revealed evolved psychological traits common to all humans, never mind that the test subjects were nearly always from the industrialized West. Henrich realized that if the Machiguenga results stood up, and if similar differences could be measured across other populations, this assumption of universality would have to be challenged.
Henrich had thought he would be adding a small branch to an established tree of knowledge. It turned out he was sawing at the very trunk. He began to wonder: What other certainties about “human nature” in social science research would need to be reconsidered when tested across diverse populations?

It should be clear from this that most social science research is nearly useless.  A good portion of social science research consist of making college students do silly tests.  This is hardly a good cross-selection of the general US population, let alone the global population.  Once you consider that a good amount of social science research is never replicated, it would seem that a good rule of thumb is that any sort of conclusions or findings of a social science experiment is valid only for the instance of its occurrence until proven otherwise.

If, for example, a group of Harvard college students being tested in the middle of September are found to walk slower after being primed with reading words associated with old age, the only conclusion you should draw from this is that there was one instance at a specific point in time in which priming Harvard college students with certain words correlated to them walking slower than normal.  This conclusion is, necessarily quite specific, and thus quite accurate.

By adhering to this sort of rigorous standard, though, you won’t make the mistake that a lot of social scientists have in extrapolating the characteristics of a group performing a very specific task in a very specific situation to the rest of humanity.  Weirdly, this will make you a better scientist than most of the people who are paid to do science.