The Mysterious Decline Effect

I've received a number of emails about my recent article on scientific replication and the decline effect, so I thought I'd try to answer a few of the more common queries. For those who aren't subscribers to* The New Yorker*, and don't want to buy the issue (but you should: Joyce Carol Oates' has a really beautiful piece), I've pasted in a sample section of the article below:

In 1991, the Danish zoologist Anders Møller, at Uppsala University, in Sweden, made a remarkable discovery about sex, barn swallows, and symmetry. It had long been known that the asymmetrical appearance of a creature was directly linked to the amount of mutation in its genome, so that more mutations led to more “fluctuating asymmetry.” (An easy way to measure asymmetry in humans is to compare the length of the fingers on each hand.) What Møller discovered is that female barn swallows were far more likely to mate with male birds that had long, symmetrical feathers. This suggested that the picky females were using symmetry as a proxy for the quality of male genes. Møller’s paper, which was published in Nature, set off a frenzy of research. Here was an easily measured, widely applicable indicator of genetic quality, and females could be shown to gravitate toward it. Aesthetics was really about genetics.

In the three years following, there were ten independent tests of the role of fluctuating asymmetry in sexual selection, and nine of them found a relationship between symmetry and male reproductive success. It didn’t matter if scientists were looking at the hairs on fruit flies or replicating the swallow studies—females seemed to prefer males with mirrored halves. Before long, the theory was applied to humans. Researchers found, for instance, that women preferred the smell of symmetrical men, but only during the fertile phase of the menstrual cycle. Other studies claimed that females had more orgasms when their partners were symmetrical, while a paper by anthropologists at Rutgers analyzed forty Jamaican dance routines and dis- covered that symmetrical men were consistently rated as better dancers.

Then the theory started to fall apart. In 1994, there were fourteen published tests of symmetry and sexual selection, and only eight found a correlation. In 1995, there were eight papers on the subject, and only four got a positive result. By 1998, when there were twelve addi- tional investigations of fluctuating asymmetry, only a third of them confirmed the theory. Worse still, even the studies that yielded some positive result showed a steadily declining effect size. Between 1992 and 1997, the average effect size shrank by eighty per cent.

And it’s not just fluctuating asymmetry. In 2001, Michael Jennions, a biologist at the Australian National University, set out to analyze “temporal trends” across a wide range of subjects in ecology and evolutionary biology. He looked at hundreds of papers and forty-four meta-analyses (that is, statistical syntheses of related studies), and discovered a consistent decline effect over time, as many of the theories seemed to fade into irrelevance. In fact, even when numerous variables were controlled for — Jennions knew, for instance, that the same author might publish several critical papers, which could distort his analysis—there was still a significant decrease in the validity of the hypothesis, often within a year of publication. Jennions admits that his findings are troubling, but expresses a reluctance to talk about them publicly. “This is a very sensitive issue for scientists,” he says. “You know, we’re supposed to be dealing with hard facts, the stuff that’s supposed to stand the test of time. But when you see these trends you become a little more skeptical of things.”

Question #1: Does this mean I don't have to believe in climate change?

Me: I'm afraid not. One of the sad ironies of scientific denialism is that we tend to be skeptical of precisely the wrong kind of scientific claims. In poll after poll, Americans have dismissed two of the most robust and widely tested theories of modern science: evolution by natural selection and climate change. These are theories that have been verified in thousands of different ways by thousands of different scientists working in many different fields. (This doesn't mean, of course, that such theories won't change or get modified - the strength of science is that nothing is settled.) Instead of wasting public debate on creationism or the rhetoric of Senator Inhofe, I wish we'd spend more time considering the value of spinal fusion surgery, or second generation antipsychotics, or the verity of the latest gene association study.

The larger point is that we need to be a better job of considering the context behind every claim. In 1952, the Harvard philosopher Willard Von Orman published “The Two Dogmas of Empiricism.” In the essay, Quine compared the truths of science to a spider’s web, in which the strength of the lattice depends upon its interconnectedness. (Quine: "The unit of empirical significance is the whole of science.”) One of the implications of Quine's paper is that, when evaluating the power of a given study, we need to also consider the other studies and untested assumptions that it depends upon. Don't just fixate on the effect size - look at the web. Unfortunately for the denialists, climate change and natural selection have very sturdy webs.

Question #2: You never mention the question of scientific fraud. Why not?

Me: Because these biases are not fraud. We sometimes forget that science is a human pursuit, mingled with all of our flaws and failings. (Perhaps that explains why an episode like Climategate gets so much attention.) If there's a single theme that runs through the article it's that finding the truth is really hard. It's hard because reality is complicated, shaped by a surreal excess of variables. But it's also hard because scientists aren’t robots: the act of observation is simultaneously an act of interpretation.

In The New Yorker article, I discussed the way our scientific observations can be shaped by our expectations and desires. (As Paul Simon sang, "A man sees what he wants to see and disregards the rest.") Most of the time, these distortions are unconscious - we don't know even we are misperceiving the data. However, even when the distortion is intentional it's still rarely rises to the level of outright fraud. Consider the story of Mike Rossner. He's executive director of the Rockefeller University Press, and helps oversee several scientific publications, including The Journal of Cell Biology. In 2002, while trying to format a scientific image in Photoshop that was going to appear in one of the journals, Rossner noticed that the background of the image contained distinct intensities of pixels. “That’s a hallmark of image manipulation,” Rossner told me. “It means the scientist has gone in and deliberately changed what the data looks like. What’s disturbing is just how easy this is to do.” This led Rossner and his colleagues to begin analyzing every image in every accepted paper. They soon discovered that approximately 25 percent of all papers contained at least one “inappropriately manipulated” picture. Interestingly, the vast, vast majority of these manipulations (~99 percent) didn’t affect the interpretation of the results. Instead, the scientists seemed to be photoshopping the pictures for aesthetic reasons: perhaps a line on a gel was erased, or a background blur was deleted, or the contrast was exaggerated. In other words, they wanted to publish pretty images. That's a perfectly understandable desire, but it gets problematic when that same basic instinct - we want our data to be neat, our pictures to be clean, our charts to be clear - is transposed across the entire scientific process.

Question #3: You end the article with the following lines: "Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true." What are you talking about? Are you some sort of Derridean post-modernist, trying to turn publication bias into an excuse to not believe in anything?

Me: Not at all. One of the philosophy papers that I kept on thinking about while writing the article was Nancy Cartwright's essay "Do the Laws of Physics State the Facts?" Cartwright used numerous examples from modern physics to argue that there is often a basic trade-off between scientific “truth” and experimental validity, so that the laws that are the most true are also the most useless. “Despite their great explanatory power, these laws [such as gravity] do not describe reality,” Cartwright writes. “Instead, fundamental laws describe highly idealized objects in models.” The problem, of course, is that experiments don’t test models. They test reality.

Cartwright’s larger point is that many essential scientific theories – those laws that explain things - are not actually provable, at least in the conventional sense. This doesn’t mean that gravity isn’t true or real. There is, perhaps, no truer idea in all of science. (Feynman famously referred to gravity as the "greatest generalization achieved by the human mind.") Instead, what the anomalies of physics demonstrate is that there is no single test that can define the truth. Although we often pretend that experiments and peer-review and clinical trials settle the truth for us - that we are mere passive observers, dutifully recording the results - the actuality of science is a lot messier than that. Richard Rorty said it best: "To say that we should drop the idea of truth as out there waiting to be discovered is not to say that we have discovered that, out there, there is no truth." Of course, the very fact that the facts aren't obvious, that the truth isn't "waiting to be discovered," means that science is intensely human. It requires us to look, to search, to plead with nature for an answer.