Note: I first published this on a now-defunct blog in January 2011
But now all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.–Jonah Lehrer
Lehrer’s recent article on the decline effect, where initially strong and robust results shrink over time, has caused quite a stir (see here, here, here, and here). It’s similar to Freedman’s Atlantic profile of Ioannidis in that it identifies some allegedly deep problems in the structure of science.
As both Lehrer and Freedman note, the fight for funding, the bias against null results, the drive for more publications, a desire for personal glory, and conflicts of interest all make the practice of science quite a bit different from how it’s usually portrayed. Both, however, also note that there may be something fundamentally amiss here. These relatively straightforward explanations only go so far. It appears we can’t quite determine what’s really going on, and Lehrer’s subtitle asks: “Is there something wrong with the scientific method?”
I think it’s important to remember that the scientific method (TSM) was first used to solve what are, conceptually at least, fairly simple questions. Planetary motion, as beautiful, intricate and awe-inspiring it may be, is solved by high-school students every day before lunch recess. After all, it’s just giant balls circling each other. Even quantum mechanics and relativity can be described almost perfectly with existing mathematics, and it’s usually first taught to college sophomores. Penicillin was discovered in a petri dish, where we can reliably control most variables. And so on.
None of this is meant to diminish Newton, Einstein and Fleming, or to trivialize their accomplishments. They were geniuses, and we are forever indebted to them. But it appears to me that what we consider the canonical scientific accomplishments were often fairly discrete questions. If you can effectively use mathematics (as you can in much of physics), or if you can control all the relevant variables (physics, molecular biology), the problem is tractable. But determining the effect of Vitamin E on public health is a different beast altogether. After all, what we call health is really a combination of traditional medical science, economic status, level of stress, culture, pollution, and who knows what else. TSM may never be able to disentangle all these effects regardless of how many perfect, unbiased studies we run. From Freedman’s article:
But even if a study managed to highlight a genuine health connection to some nutrient, you’re unlikely to benefit much from taking more of it, because we consume thousands of nutrients that act together as a sort of network, and changing intake of just one of them is bound to cause ripples throughout the network that are far too complex for these studies to detect, and that may be as likely to harm you as help you.
And so asking what’s wrong with TSM kind of misses the point. There’s nothing really wrong with it. It’s that TSM, this crowning achievement of human intellect, the best tool ever developed to investigate the natural world, may just not be as powerful as we want it to be. Even if we disclose conflicts of interest, release all collected data (not just the pretty ones), publish more null results, and quadruple funding, some problems will be beyond our reach. There are ultimately some things in heaven and earth not dreamt of in our philosophies.