Who’d have thought that medical science can be used to examine how good scientists are?
Large double-blind randomized control trial (RCTs) are about as good as it gets for figuring out whether some new medical treatment actually works. In RCTs, patients are randomly allocated to either placebo treatments or the experimental treatment, and if it is all done well (blinded, large scale, properly randomized), an answer emerges which is a great deal more reliable than can be had any other way. But at the heart of RCTs is a paradox. Only treatments which scientists think will work are tested in these expensive trials. So what patient in their right mind wants to be in the control arm? Everyone should want to get the new treatment – the scientific bet is on the new treatment. Indeed, if medical science is any good, the very trials needed to test large-scale efficacy and safety are unethical!
Fortunately, it turns out that medical science is actually not that good. Analysis of 50 years of U.S. cancer RCTs have shown that that most of the time, the new experimental treatments do nothing at all, but that when they do, they are
as likely to harm as help. Consequently, the trials are – magically – ethical (whew). On average, you are as well off getting the placebo as you are getting the fancy experimental treatment.
That’s good news for medical science in one sense, but appalling news in another. It means that despite all our knowledge of biochemistry, cell and molecular biology, all the test-tube experiments, all the animal studies, all the small-scale patient trials, all the theory and experience in the world, when it comes to identifying treatments that will work, we’re no better than a coin toss. Importantly, only around 20% of the trails are breakthroughs. There is 80% waste (or harm) in the process, and that is excluding all the ideas that never survived to get to RCT stage — say ten times as many again?
Weeks after I taught this to the class, I still find it staggering. I think my science has a better hit rate than 2% — but then, there is no way to quantify success and failure in my business. In perhaps the one area of science where success rates can be quantitified, it turns out we are staggeringly poor at discovery.
Of course, all that waste has nonetheless powered enormous improvements in cancer outcomes. And it has certainly been way more successful at cancer treatment than centuries of exploration by other approaches to knowing (think religion, witch doctors, homeopathy, philosophy, and non-evidence based medicine). But still. It just shows how much we really are groping in the dark.