Who to Trust? Difficulties in Science Journalism

As a father of a three year old and as a scientist, the article this year that really stuck with me and encapsulated my frustration with medicine (and “medicine” and medical reporting) was this one in Slate about giving “alternative” medicine to your children.  I hope its popularity helps a lot of people avoid harming themselves and their family with “natural” and “alternative” remedies (pass it on!).
But it’s not the “chemophobia is bad for you” thesis that struck me so much as the way that it illustrates how easily we come to swear by unproven and even untested remedies and dietary / medical / parenting advice.  
But even avoiding “chemophobia” and the naturalistic fallacy is not enough.  When there is no clear scientific consensus on something, it’s easy even for scientists to grab on to something that is pitched just perfectly to make you think “yeah, that’s probably right, I’ll go with that.”  I certainly follow unproven diets and parenting ideas: this article in the New York Times Magazine led me to dramatically decrease my added sugar intake to almost nothing (I’ve eased up now that I’m in the habit of avoiding it) and this book about baby sleep habits has ruled my daily schedule for over three years now.
But I feel I do this knowing and even expecting that these sources are imperfect, and maybe even wrong;  I have various rationalizations for following them regardless.  Most people don’t apply a scientist’s skepticism to big decisions like that, and that leads to things like people giving unknown doses of powerful, untested drugs with unknown side effects to their children and then bragging about it in the New York Times Magazine.
Part of my frustration is that science journalism is hard, and as a result, a lot of it is pretty bad.  Many science journalists can’t distinguish between firmly established fact and plausible but untested hypothesis.
To give an example of how even good journalists can fall prey to this, the (excellent) Lee Billings had this twitter exchange with (NASA/Library of Congress Astrobiology chair) David Grinspoon upon the launch of Maven:

Here is a very good reporter on astrobiology basically getting caught mistaking a plausible but untested story about Mars’s atmosphere for a well-established conclusion. And if Lee Billings can get it wrong about his area of expertise, then imagine the poor science journalists tasked with covering all of health issues reporting on a new scientific development in diet or medicine, interpreting puffed-up press releases of research that is notoriously plagued with publication bias, sloppy statistics and subsample analysis, and uncontrolled experiments.

For instance, it’s common to read that a certain form of cancer or other disease is on the rise.  But it’s actually hard to do a proper study of this: old people tend to get cancer and other diseases.  Cancer could be on the rise because of environmental effects (bad) or because medicine is making people live long enough to get cancer (yay science!).  How can a science journalist be sure to cover bases like those on every story they cover?
Similarly, once often reads that modest liquor intake is salubrious, in that people who drink lightly and occasionally have better health than tee-totallers.  But how many of those studies were meta-analyses or sub-sample analyses?  Did they consider why the tee-totallers didn’t drink?  If they are Mormon or Muslim they probably represent a different population from the drinkers demographically and dietarily, so aren’t a good control sample.  If they don’t drink because they have liver damage or are recovering alcoholics then their poorer health is even easier to explain.  How did the studies control for these effects?
Even worse, how did these studies control for all of the effects that haven’t been considered yet?
In short, medicine and journalism are hard, and we all need to be skeptical about medical results, especially when interpreted through the lens of journalism.  But also, medicine is awesome, and makes our lives better, and we should heed its advice, which it communicates through good and important journalism.  Trans-fats are a good example of the process apparently going very right.
It makes me glad I’m an astronomer, and not a medical researcher or journalist:  it’s much easier to do well, and no one really gets hurt when I’m wrong.
So, have a Happy New Year, lay off the sweets during the celebration, and don’t let your younger kids stay up to see the ball drop.  Or maybe not, what do I know? ;)

One thought on “Who to Trust? Difficulties in Science Journalism”

  1. Geoff Marcy

    This was a beautiful piece, Jason. Thank you. I enjoyed that you respect the challenges of knowing and reporting what might be right from wrong. And you captured that message by the very words of your advice, with due humility. Well done.

Comments are closed.