Journalism has been in trouble since… well maybe since forever but it certainly feels more precarious since the World Wide Web destroyed most print journalism as a viable stand-alone business. The New York Times and Washington Post have hung on with quality journalism, but not without moving to a heavy online presence and worrying a lot about generating “clicks” (at least on the business side).
One way to survive is to generate lots of those clicks, and that means “clickbait”—provocative headlines that dare you to ignore them. Whether you find this to be outrageous or just a fact of business probably depends on how exposed you have been to it. I imagine most journalists take it as a given that headlines and ledes must generate clicks and scrolls in order for them to have jobs, but that what really matters is the meat of the article.
It would be nice if it weren’t so, especially when it’s your science being hyped. Yesterday I wrote about Sabine Hossenfelder’s Forbes article and along the way defended it against accusations that it was “sloppy journalism” and representative “the frenzied sensation-driven nature of mainstream publishing” mostly by linking to it, quoting it, and being incredulous at those descriptions. Some have disagreed with that assessment:
Creswell paper didn't contain claim that 'it could just be noise', rather they said 'noise around this signal is not entirely understood'.
— TomD_HD (@TomD_Hann) June 23, 2017
As I believe I said, the lede is quite misleading. A set up "big experiment may be wrong", followed by "but probably not" is risky sci comm
— Jonathan Pritchard (@jr_pritchard) June 22, 2017
First of all let me sympathize: it is mortifying to see your research in print in a way that sensationalizes it.
Next, let me point out that this is not even close to yellow journalism. I know yellow journalism, and Forbes, senator, is no yellow journalism.1
So maybe it’s because I’ve seen a lot worse and I’ve written for popular outlets and my student Kimberly Cartier wrote a thesis about science communication that I’ve become inured to the way that headlines beg for clicks, even if they don’t represent the meat of the article well.
But I think in this case the Forbes article is actually not in that camp. With the benefit of hindsight or insider knowledge, it might look overblown, but consider:
- At the time it was written, there was no LIGO response to go on.
- Hossenfelder contacted LIGO and they had no formal response—and said they planned none!
- Hossenfelder then got a third party to comment, and they found “the results “quite disturbing” and hope[d] that the collaboration will take the criticism of the Danes to heart. ‘Until LIGO will provide clear scientific (!) explanation why these findings are wrong, I would say the result of the paper to some extent invalidates the reliability of the LIGO discovery.’”
- We all know that the scientific literature is often understated, especially when suggesting someone is wrong. An abstract that concludes “A clear distinction between signal and noise therefore remains to be established in order to determine the contribution of gravitational waves to the detected signals.” has a pretty clear meaning: the GW detections are in jeopardy.
- After all, the strongest signal was 5.1-sigma. If there is any reason to think the noise is underestimated then the signal drops below the 5-sigma level. Now I appreciate that the difference between “significant” and “not significant” is not Itself statistically significant,” but the “New York Times threshold” is actually the one place where which side of 5-sigma you are on absolutely matters!
Given all these items, I think the headline and fInal introductory sentence of the Forbes article is fine:
Was It All Just Noise? Independent Analysis Casts Doubt On LIGO’s Detections
…But what if there wasn’t a signal at all, but rather patterns and correlations in the noise that fooled us into believing we were seeing something that wasn’t real? A group of Danish researchers just submitted a paper arguing that the celebration might have been premature.
Especially given the 5.1 sigma issue, I think that’s totally fair. The paper did cast doubt, none of Hossenfelder’s followup investigations dispelled that doubt, and “celebration might have been premature” is an accurate description of the implications if the detection was actually only, say 2.5-sigma (and Creswell et al. implied it could be much lower than that, IMO).
Anyway, YMMV on this. We can agree that it would be nice if popular journalism were as sober as The New Yorker and as popular as cat memes, but I think we can also agree that that will not happen.
The only practical solutions I see to the larger problem are:
- Get scientists better at science communication.
- Appreciate those that are good at it and see them and the work they do as an important part of our profession (the aforementioned Kimberly Cartier included a chapter on this as part of her PhD thesis in Astronomy & Astrophysics—I hope in the future this will not seem novel).
- Make sure that when our work pops up in the media we know how to manage it. Especially a large, high-profile project like LIGO should have1 team members (or access to professionals) that know how to quickly manage stories.
After all, if LIGO had told Hossenfelder that it was planning a response that completely addressed the Creswell et al. paper and showed it was based entirely on poor analysis, her article would have had a different conclusion and different tone (if it had been written at all!). They had an opportunity to get in front of it and shape the Forbes article, but they did not.
But it all turned out OK: their eventual (though unofficial) response is solid and succeeded because they put it on Sean Carrol’s popular and respected blog instead of waiting for the peer review process. Now the story can be “nothing to see here, move along” and, as a bonus, the referee of the Creswell et al. paper has a great template for their review. Double bonus: now the world understands LIGO better!
1For the kids: Bentsen-Quayle.
[Edit: Hannam responds (click to expand):
Poor papers are a dime a dozen. Experts recognise them as such and ignore them. (Life is too short.) 1/n https://t.co/He52JNP7j4
— Mark Hannam (@MarkDHannam) June 23, 2017
]
1I don’t mean to imply LIGO doesn’t have such people—on the contrary I mean that the (by all accounts very good) people it has are essential.
Compare the bitter and improperly censoring tone, with repetitive propganda-like cliche usage, of the dishonest and uncorrected Mark Hannam blog response to the Creswell et al. https://fictionalaether.blogspot.com/2017/06/the-irresistible-allure-of-controversy.html
…which was complementary to Sean Carroll’s blog, and also deliberately omitted that it was acknowledged by LIGO that Ian Harry had made the error, of course also assuming Gaussianity and Gaussian noise were colored by quantum and processing artifacts as a kind of form of retroductive evidence (a categorical fallacy in this context). This is the most recent update from representatives of the falsification approach practiced by the Danish group.
http://www.nbi.ku.dk/gravitational-waves/hannover-presentation.html
Also, see this https://www.facebook.com/fulguritics/posts/10216276647856824
this https://www.facebook.com/fulguritics/posts/10216256355029516
and this https://www.facebook.com/fulguritics/posts/10215562542524637?hc_location=ufi
I have maintained an observation protocol relating multiproxy space physical data and geophysical data with unusual coherent lightning behavior, having already been involved with this study as the GW150914 was announced. I immediately suspected that, like some theorists have argued, thunderstorm activity may be modulated in subtle ways by passing transient gravitational waves. The appearance of critical spatial correlations in active thunderstorm cell clusters observed independently during a study of magnetospheric sawtooth injection events not correlated with concomitant substorms (reported as rare events, with ordinary sawtooth oscillation systems few than 12/yr for solar cycle 23) prompted that I make any possible effort to exclude these possible and known noise sources, attempting to produce correct signal lags, LIGO parameter estimations, and waveforms from proxy and dummy data. I am currently developing robust and non-parametric signal processing and characterization regimes, utilizing geographic models and signal reconstruction and inversion tests using compound spectral density analysis of many versions of the same data. I am not a denier of GR or GW, but I am not comfortable with a mono-theoretical experiment of this scale that has not, after decades, found any formal or even physical system to classify unwanted noise transients (“glitches”). Thanks for your objectivity and time. Contact me for code and workbooks if you are interested. The analysis is much more extensive than these particular samples I have linked publicly.
Thomas, I think the latest from the Danish team contradicts your assertion that they are not calling the GW detection into question:
“if we are to conclude reliably that this signal is due to a genuine astrophysical event, apart from chance-correlations, there should be no correlation between the “residual” time records from LIGO’s two detectors in Hanford and Livingston”
Logically, this means that if there *are* such correlations, then we are *not* to conclude reliably that the signal is due to a genuine astrophysical event.
http://www.nbi.ku.dk/gravitational-waves/
So I think this supports the propriety of the Forbes headline.
Also, you write:
“It would have been really responsible of Forbes and Sabine H. to look for an independent expert assessment of whether the Creswell paper had any merit or chance of being correct. ”
But she did! She had an off-the-record source call the results “‘quite disturbing” and hope that the collaboration will take the criticism of the Danes to heart.”
Thomas Dent replies:
Two serious problems with the discussion here. First the description of communications with LIGO : “She, very responsibly, went to the LIGO team for a response: ..”
As a (junior) LSC member I can tell you this is factually incorrect, Sabine H. did not contact LSC management at any time for a quote or reaction. The quote was from an internal email discussion of what LSC members *could be advised to say* if they were asked about this paper – an email which she obtained through informal contacts within LSC.
(BTW, how could anyone describe an email as ‘annoyed’ .. ? oh that’s right, you can’t deduce tone from emails. So that couldn’t be anything but bias on the part of the writer.)
Eavesdropping an internal discussion of how people *might* respond to a paper is very different from actually contacting the collaboration spokesperson, telling them you are writing an article for a major news website and asking for a response. A real journalist would have done that.
The second real problem is with the title and introductory paragraphs of the Forbes article, which are, yes, simply inaccurate : sensationalized misrepresentations of the content of the paper, and Mark was correct to complain about them.
The paper did NOT say ‘it could just be noise’: it said, more or less, ‘we think the noise around the detected GW signal is not entirely understood’. Read it! So the title of the post was a (probably deliberate) distortion and exaggeration. Given the damage done in the first few sentences, the reversal of tone in the last paragraph – ‘oh actually I think the paper’s probably wrong but LIGO still has to prove themselves right regardless’ – is hardly sufficient to achieve balance.
It would have been really responsible of Forbes and Sabine H. to look for an independent expert assessment of whether the Creswell paper had any merit or chance of being correct. Without a commitment to quality control both science and science journalism are close to useless.