Megan McArdle:

“Conflict epidemiology,” the study of war’s health effects, is by its nature an inexact science. War and anarchy are not friends to careful, by-the-book research. We have little idea how many people now live in Iraq; ascertaining the number who have died there is a tall order. And huge disparities in death estimates are not unique to the conflict in Iraq; cluster sampling, the best-regarded survey technique for use in war-torn places, has produced estimates in other conflict zones, such as Darfur, that vary by factors of three or more.

All casualty studies have problems. But the Johns Hopkins study’s methodology was particularly troublesome. The number of neighborhoods the team sampled was just above the minimum needed for statistical significance, and the field interviewers rushed through their work. The interviewers were also given some discretion over which households they surveyed, a practice generally regarded as unwise. And though such latitude calls for closer-than-normal supervision of field interviewers, the Johns Hopkins team seems to have provided little. Any of these choices can be defended because of the dangers, and the authors have said as much, claiming, basically, that this was the best they could do in a bad situation.

But that raises an unwelcome question: If this is the best we can do, should we be doing this at all? Cluster sampling was developed for studying vaccination; it has never been validated for mortality. Because of the wide variance in the estimates it produces, some researchers are now questioning its usefulness.

Yet though its compromises made it particularly unreliable, the Lancet study remains the most widely known. Its conclusions were the earliest and most shocking of the scientific estimates and thus generated enormous media attention. The more-careful counts that followed prompted fewer, and less prominent, articles. There’s little doubt that the larger number will live on for years in the writings of antiwar activists. But the rest of us, too, were influenced by it, perhaps more than we realize. We will have to live with its legacy.

Most data create what cognitive scien- tists call “anchoring effects”: we fixate on numbers we’ve heard, even if they’re arbitrary or wrong. In one 1970s experiment, Amos Tversky and Daniel Kahneman (whose work won a Nobel Prize) famously picked a number at random in front of their subjects, by spinning a wheel, and then asked them to guess whether the percentage of African nations in the UN was higher or lower than that number. Next, they asked for a hard estimate of the actual percentage. The higher the random number, the higher the final estimate tended to be, even though the first number had been obviously irrelevant.