1.

Publication Bias Against the Null Result

The existence of a publication bias against null or negative results was first described in 1959,² and this bias has stayed the same³ or gotten worse since then.⁴ Many studies have shown that the vast majority of published scientific papers show positive results, that input A does in fact affect output B in the desired way. Negative results suffer from the “file drawer” effect: a study that finds no impact of A on B will likely be filed away in the researcher’s desk drawer rather than published in a peer-reviewed journal.⁵ This leads to an incorrect impression that such experiments have never been tried.

There are three potential reasons for the existence of such a publication bias: editorial policy, reviewer bias, and author submission bias. While there may be some journals that actively discourage the publication of negative results through their editorial policy, such journals are probably the exception and certainly JM³ is not among them. Reviewer bias is probably more common, since reviewers are tasked with evaluating the significance of a manuscript and there is often an unstated assumption that positive results are more significant than negative results.

Still, I think submission bias accounts for a majority of the publication bias. Authors, either anticipating a reviewer bias or having a bias for positive results themselves, are much more likely to submit a manuscript that contains positive results than negative results. A journal cannot publish a paper that demonstrates a null result if that paper is never submitted. The reasons for these biases are probably rational: positive results generally attract more readers and citations. The undesirable consequences, however, can be significant.

2.

Consequences of a Publication Bias

There are two major consequences of the publication bias against the null result, both unpleasant in their own way. The first is wasted effort. As I mentioned, most researchers are looking for positive results: they are trying to reduce the leakage current of a CMOS transistor, increase the Q-factor of a MEMS device, or reduce the roughness of a lithographically patterned feature. They try many different approaches, testing the effectiveness of many different variables. Most of the approaches don’t work, but a few yield positive results. If the publication bias is at work, only the positive results are published, and the fact that certain experiments led to null or negative results remains unmentioned.

If readers remain unaware of these negative results, they are more likely to repeat these experiments in their own efforts to find positive results. The consequence is unnecessary waste. A completely valid and potentially important scientific outcome, that input A does not impact output B, is not published and so does not join the collective knowledge of the community. And the search for positive outcomes proceeds more slowly as a result.

The second consequence of the publication bias against null results is more insidious: it increases the likelihood that published results are wrong. In some cases, entire fields of study (such as extra sensory perception, ESP) publish only spurious positive results⁶ (since a negative result, showing no evidence for ESP, would be unlikely to be published). But leaving aside such extreme cases, there is evidence that the publication bias against the null result leads to significant publication of spurious positive results in most or all fields, as John Ioannidis has persuasively claimed in his provocatively titled essay “Why Most Published Research Findings Are False.”⁷

Consider twenty researchers all independently trying to see if input A affects output B. If A really has no impact on B, then one out of the twenty researchers will likely produce a spurious positive result to a 5% significance level ($\alpha =0.05$) by pure chance. This will not cause any problems if all twenty researchers publish their results. But if the nineteen null findings remain unpublished (the file drawer effect) and the one spurious positive result is published, readers will very reasonably assume that the results in the one published paper are representative of all studies and are likely to be true. The publication bias against the null result naturally leads to a degradation of the overall quality of published research as a whole.⁸

But science is supposed to be self-correcting, imbued with a “trust, but verify” mentality. Replication of results by other researchers should ferret out these spurious positive findings, eventually leading to sound conclusions. But “eventually” can be a long time. Further, there is some evidence that most scientific studies are never replicated, so that bad results can linger in the collective consciousness of the scientific community for a very long time.² The “publish or perish” mentality in academia, coupled with a publication bias against the null result, means that the scientific community often rewards impact and quantity over reproducibility and quality. Few scientists seem willing to devote significant time and resources towards replication of others’ results.

3.

A Modest Proposal

I have no illusion that this editorial, or any editorial policy I may try implement at JM³, will change the current culture of science and make the bias against the null result disappear. Authors, reviewers, and even editors at JM³ are likely to continue to favor positive results over null or negative results. So here is my modest proposal to help mitigate the negative impacts of a publication bias against the null result.

Authors, when writing up your paper and emphasizing the positive results that you think are most important, please don’t forget the negative or null results that you found along the way. Include a few sentences about the variables you tried that didn’t produce the desired effect. Show a graph of the data that demonstrates no significant effect, if for nothing else than to compare to the graph of data that does demonstrate the desired effect. Think about all the dead-ends and blind alleys that you went down in your search for a solution to your problem, then warn the rest of us about them. Consider the null result as a valid and important scientific discovery, and add it to your paper of positive results.

Reviewers and editors, don’t recommend that null results be deleted from a paper just because they are null results. While you may always consider the positive result to be more significant, don’t automatically think that a null result is not important. Consider all the wasted effort that can be avoided if just a few paragraphs of a paper are devoted to those null results that are almost always lurking around every scientific study.

Null results are an important part of science. I hope they will be an important part of JM³ as well.