Should I publish negative results or does this ruin my career in science?

Young scientists often produce negative results. All experiments were done correctly – but there was no difference between test and control. They get conflicting advice from supervisors and ethicists. Some say that publishing negative results is a waste of resources and ruins their careers. Others say that ‘not publishing negative results is unethical’ and promotes the reproducibility crisis. What should young scientists do in such a situation?

NEGATIVE RESULTS ARE FRUSTRATING AND DEMOTIVATING

In my experience as a supervisor, the biggest problem of negative findings is the demotivating effect on young scientists. During a normal PhD, most early stage researchers experience an emotional dip anyway – approximately after two years.  This is half the duration of a standard PhD contract. It is a recurring pattern I have seen often.

In addition, they are probably worried about their future. On the other hand, it may also relieve the student because they may have felt for a long time that their project is going nowhere. These mixed feelings do not make the situation easier. It is the responsibility of the supervisor to develop a new project and to support the PhD student emotionally in this difficult situation.

“PUBLISHING NEGATIVE RESULTS IS A WASTE OF RESOURCES”

appropriate journal and after several rejections it ends up in a journal with a much lower impact factor than expected at the beginning of the experiments. Consistently, negative studies are considered less prestigious if and when they do come out.

UNEXPECTED NEGATIVE FINDINGS CAN BE VERY INTERESTING – BUT ARE OFTEN COSTLY

High impact journals may be interested in negative studies when they destroy a long-held paradigm or when a new method is used to show that most previous studies are flawed. Following up on a negative story is always considered a risk because the research group may invest substantial resources (time, money, energy) without improving the quality of the paper and the final impact factor.

Reviewers may tend to ask multiple additional controls to make sure that the negative results are not just an effect of technical mistakes. Therefore, the opportunity costs are high (“Doing this study means you are not following up on potentially more promising data”).

“PUBLISHING NEGATIVE RESULTS RUINS YOUR CAREER”

However, impact factors are still broadly used to evaluate the performance of single scientists, departments and institutions (read more here: 10 simple strategies to increase the impact factor of your publication). 

Thus, a responsible supervisor will always aim for journals with a high impact factor and will have the tendency to abandon projects without exciting results. Unfortunately, this behavior – as understandable as it is – may be one of the biggest problems in science.

“NOT PUBLISHING NEGATIVE RESULTS IS UNETHICAL”

Many colleagues do not find it a big deal to abandon projects without promising result. “Fail faster” is a common credo which means to screen for dramatic effects (for example of a treatment, a drug, a genetic intervention etc.) and leave the less dramatic results untouched.

The big problem is that this knowledge is lost because all these experiments disappear and many other scientists may repeat the same or similar experiments because these results are not documented and are not publicly available. As a consequence, a lot of time, effort and taxpayer’s money is wasted due to unnecessary repetitions because negative or less-than-dramatic findings are unreported.

If you care for scientific progress, publication of negative results is a must. Negative studies may challenge existing paradigms and enhance progress by stopping further investment in scientifically barren topics, decreases the use of animals in experiments, and focuses research in more fruitful areas (Boorman et al., 2015).

PUBLICATION BIAS AND THE REPRODUCIBILITY CRISIS

More than 70% of researchers in a Nature’s survey of 1 576 researchers have tried and failed to reproduce another scientist’s experiments (Baker et al., 2016). This so-called reproducibility crisis has many reasons (see reviews in Jarvis & Williams, 2016; Begley & Ioannidis, 2015).

One important reason for reproducibility problems is positive-results bias, a special form of publication bias. Positive-results bias is just a fancy term for the tendency described above: when authors are more likely to submit, or editors to accept, positive results than negative or inconclusive results (Sacket et al., 1979). 

Briefly, publishing only the positive results and filing away the negative results produces a skewed view of reality, results in unnecessary repeats of experiments already done, wastes a lot of taxpayer’s and industry money, may lead to detrimental therapies and many frustrated scientists among many other unwanted outcomes. 

Reproducibility issues may even contribute to a growing skepticism regarding the integrity and relevance of all biomedical research (Jarvis & Williams, 2016).

IT IS NOT THE TASK OF YOUNG SCIENTISTS TO PUBLISH NEGATIVE RESULTS.

After many years of struggling with this question I came to the conclusion that it is *not* the task of young scientists to publish negative results – in the current system. They still have to pursue their career and – as explained above – publishing negative results may have quite negative effects on their careers.

CONTROVERSIAL ADVICE: LEAVE IT TO THE OLD GUYS!

The arguments listed above bring every scientist to a difficult situation. Young scientists seem to have to choose between their career and ethical behavior. Supervisors have to give good advice or leave the young scientists alone with this decision.

However, right now there are no widely accepted and widely known procedures how to handle negative findings which are not used for a publication (find some suggestions below). Therefore, currently the best advice is:

Do not publish negative results as a young scientist. Leave it to the senior scientists who already have a successful career and can afford it to publish negative findings for the sake of good science!

It is very important to note that I do *not* suggest selective reporting. Selective reporting is a special case of reporting bias, resulting in the incomplete publication of analyses performed in a study that leads to the over- or underestimation of treatment effects or harms. Selective reporting is scientific misconduct. In contrast, I advise young scientists not to waste their time, grant money and energy on studies with negative results.

In the meantime, let us work on better procedures and rules to avoid “scientific waste” and substantially reduce unnecessary repeats of the same unpublished experiments.

HOW TO BEHAVE ETHICALLY AND HANDLE NEGATIVE RESULTS CORRECTLY?

My advice to leave the publication of negative results to the old guys (who already have a successful career) comes with a price. We accept the publication bias for the sake of young scientist’s careers. This promotes the reproducibility crisis.

The old guys cannot be trusted either to take over their responsibility to publish negative results because they do not get rewarded or even worse they even get punished by their institutions and funding agencies which are strongly focused on scientific output which is mostly quantified in impact factors, citations and grant money.

In other words:  The scientific community gets what it incentivized. There are many ideas to improve the current science system (see for example: Begley & Ioannidis, 2015) but most initiatives are still at an embryonic stage.

To my knowledge there are no internationally accepted rules on how to handle negative results. A non-representative survey among 10 young professors revealed that only one had a rough idea where to store unpublished data which may be of interest to other scientists. Below you find three suggestions on how to handle the problem of negative results at the level of the scientific community:

THREE STRATEGIES TO IMPROVE SCIENCE

1. REGISTRATION OF ALL STUDIES *BEFORE* THEY START

Registration of clinical trials is a widely recognized tool for facilitating complete public reporting (Zarin & Tse, 2008, Williams et al, 2010) and to counter the conflicts of interests for example of pharmaceutical companies. 

Registration of any type of study (pre-clinical, observational etc.) would dramatically increase the administrative load for researchers, institutions and funders. However, it seems to be a necessary step to cure science from positive-results bias.

For clinical trials, there is a list of international registries in the Cochrane handbook and in Williams et al., 2010 (though note that some of these registries may have been lost by now). Some of these registries already allow the registration of non-clinical studies (observational, pre-clinical etc.). 

For social sciences and other types of research, the Open Science Framework (https://osf.io) offers the possibility to pre-register studies. It is compatible with GitHub so even the paper development, textual changes, data entry, and data analyses can be tracked throughout the process.

2. SAVING INCONCLUSIVE DATA IN PUBLICLY ACCESSIBLE REPOSITORIES TO MAKE THEM AVAILABLE TO OTHER SCIENTISTS

Since publishing inconclusive data may be tedious and considered “a waste of resources” (see above) it should be very easy and a standard procedure to save the data in publicly accessible repositories. To guarantee independence these should be financed by the international scientific community via scientific societies and/or national and international funders.

Some repositories already exist – see here a list of repositories on the Nature website. Another example is arxiv.org, which is a repository for documents and papers rather than data itself, including unaccepted or unsubmitted manuscripts, powerpoint presentations, etc. Unfortunately, repositories are often costly and researchers are not incentivized to use them.

3. FUNDERS MUST OBLIGE SCIENTISTS TO PRE-REGISTER THEIR STUDIES AND TO MAKE ALL DATA AVAILABLE (FOR EXAMPLE IN A PUBLICLY ACCESSIBLE REPOSITORY OR IN A JOURNAL OF THE FUNDER)

Finally, funders should take their responsibility and provide a standardized procedure to pre-register all funded studies and to oblige the researchers to publish all their negative findings or deposit them in a public repository. Some funders have already started their own journals to publish these negative findings.

A possible incentive may be to freeze the last tranche of the funds (e.g. 25%) until data are made publicly available. Without doubts, there must be very flexible regulations because some studies are published many years after the funding period has finished.

CONCLUSIONS

The final advice is simple:

As a young researcher:

• Do *not* publish negative results, leave it to the old guys.

As a senior scientist:

• Take your responsibility and publish negative findings for the sake of science even with a lower impact factor but do not force the young scientists to do this.

All scientists:

• Make negative findings known in reviews (which can be published with a high impact factor) and in scientific talks.
• Push your institution, your scientific society and/or your funders to provide a public repository for negative results and to make it easy to make negative results publicly available without the hassle of a full publication.
• Save your negative results or inconclusive data in a public repository to make them available.
• Help to improve the system to incentivize the publication of negative findings and replication studies – for example talk to funders or higher education and health politicians – many listen to scientists to improve the current procedures.

If you have any other idea to improve the current system please leave a comment below.

ACKNOWLEDGEMENTS

A lot of thanks to Noémie Aubert Bonn who provided the information about registries and repositories and helpful comments.