by

Should I publish negative results, or does this ruin my career?

Scientists often produce negative results. All experiments were done correctly – but there was no difference between test and control. They get conflicting advice from supervisors and ethicists. Some say that publishing negative results is a waste of resources and ruins their scientific careers. Others say that ‘not publishing negative results is unethical’ and promotes the reproducibility crisis. What should young scientists do in such a situation?

Do you want this article as a PDF file? Click here.

Content

Negative results are frustrating and demotivating

Sad person representing the bad feelings of getting negative resultsIn my experience as a supervisor, the biggest problem of negative findings is the demotivating effect on young scientists.

During a typical PhD, most early-stage researchers experience an emotional dip anyway – after approximately two years.  This is half the duration of a standard PhD contract. It is a recurring pattern I have seen often.

The emotional dip is substantially worsened when the students are ambitious and work particularly hard to accomplish a great project, but do not obtain statistically significant results.

Stopping a long-term project may hurt and/or relieve

Suppose a costly project does not deliver promising results. In that case, the supervisor may conclude that the project has to be abandoned without publishing the data and alter the research plans accordingly.

In many cases, such a decision will push the young researchers into a vortex of dramatic feelings. They may perceive themselves as a failure and feel treated unfairly.

In addition, they are probably worried about their future. On the other hand, it may also relieve the student because they may have felt for a long time that their project is going nowhere. These mixed feelings do not make the situation easier. The supervisor is responsible for developing a new project and emotionally supporting the PhD student in this challenging situation.

“Publishing negative results is a waste of resources”

Many scientists believe that following up on and publishing negative results wastes resources of their research teams. Indeed, this statement has some truth because the costs and research effort in a particular study may be high compared to the final output parameters (such as the impact factor of the journal, citations, etc.) reached at the end.

Negative results are often published with a low impact factor because editors hate boring data

Dollar sign representing that publishing is a big businessPublishing is a big business – in particular, open-access publishing. Editors play a pivotal role in the perception of the value of negative results.

To increase the visibility of their journals, editors want the exciting stuff: new mechanisms, unexpected findings, and dramatic effects (“the paralyzed could walk again”), which increase citations, clicks, shares, and press coverage.

Unfortunately, negative results are often very boring. As a result, it is often difficult to publish negative results in an appropriate journal.

After several rejections, it ends up in a journal with a much lower impact factor than expected at the beginning of the experiments.

As a result, negative studies are considered less prestigious by many scientists.

Several peer-reviewed journals specifically focus on negative findings (see below). Unsurpringly, their impact factors and citation scores are rather low.

You might falsely hope that open access platforms  or OA journals might be more open to well-performed studies with negative results. In my experience, the editors of OA journals are as selective with negative studies as all the other scientific journals.

Unexpected negative data can be very interesting – but are often costly

High-impact journals may be interested in negative studies when they destroy a long-held paradigm or when a new method is used to show that most previous studies are flawed.

Following up on a negative story is always considered a risk because the research group may invest substantial resources (time, money, energy) without improving the quality of the paper and the final impact factor.

Reviewers may ask for multiple additional controls to ensure that the negative results are not just an effect of technical mistakes. Therefore, the opportunity costs are high (“Doing this study means you are not following up on potentially more promising data”).

“Publishing negative results ruins your career in science”

The words your future in a toilet representing a ruined scientific careerMany supervisors are convinced that publishing negative results will ruin the career of their PhD students as well as their own. They will spend a lot of resources on the wrong project, publish with a low impact factor, and consequently get less future funding.

Young scientists may feel that investing in a negative study will dramatically reduce their chances on the academic job market and may even force them to give up and pursue a different career.

You may correctly argue that high-impact factor publications are not absolutely necessary to become a professor and might not necessarily increase job opportunities in the industry or public sector.

Read more here:  Do I need Nature or Science papers for a successful career in science?

However, impact factors are still broadly used to evaluate the performance of single scientists, departments, and institutions.

Read more here: 

10 simple strategies to increase the impact factor of your publication)

Which bibliometric data are relevant for a research career?

Thus, a responsible supervisor will always aim for journals with a high impact factor and will tend to abandon projects without exciting results. Unfortunately, this behavior – as understandable as it is – may be one of the biggest problems in science.

Not publishing negative results is unethical”

Crossed fingers representing unethical handling of negative findingsMany colleagues do not find it a big deal to abandon projects without promising results.

“Fail faster” is a common strategy that means to screen for dramatic effects (for example, a treatment, a drug, a genetic intervention, etc.) and leave the less dramatic results untouched.

The big problem is that this knowledge is lost because all these experiments disappear, and many other scientists may repeat the same or a similar experiment because these results are not documented or publicly available.

In these instances, accessing research data from similar experiments that were never published could save much time and financial resources.

The scientific literature is awash with positive findings, which naturally paints a skewed perspective on the existing scientific knowledge. The publication of null results or negative data would add much-needed balance and also provide valuable information for systematic reviews.

As a consequence, a lot of time, effort, and taxpayer money is wasted due to unnecessary repetitions because negative or less-than-dramatic findings are unreported.

If you care for scientific progress, publication of negative results is a must. Negative studies may challenge existing paradigms and enhance progress by stopping further investment in scientifically barren topics, decreasing animal use in experiments, and focusing research in more fruitful areas (Boorman et al., 2015).

Publication bias and the reproducibility crisis

More than 70% of researchers in a Nature’s survey of 1 576 researchers have tried and failed to reproduce another scientist’s experiments (Baker et al., 2016). This so-called reproducibility crisis has many reasons (see reviews in Jarvis & Williams, 2016; Begley & Ioannidis, 2015).

One important reason for lack of reproducibility problems is positive-results bias, a special form of publication bias. Positive-results bias is just a fancy term for the tendency described above: when authors are more likely to submit, or editors to accept, positive results than negative or inconclusive results (Sacket et al., 1979). 

Briefly, publishing positive results only and filing away the negative findings produces a skewed view of reality, results in unnecessary repeats of experiments already done, and wastes a lot of taxpayer and industry money. It may lead to detrimental therapies and many frustrated scientists, among other unwanted outcomes. 

Reproducibility issues may even contribute to a growing skepticism regarding the integrity and relevance of all biomedical research (Jarvis & Williams, 2016).

It is not the task of young scientists to publish negative results.

After many years of struggling with this question, I came to the conclusion that it is *not* the task of young scientists to publish negative research outcomes – in the current system.

They still have to pursue their career and – as explained above – publishing negative results may have quite negative effects on their careers.

Controversial advice: Leave it to the old guys!

Old scientist representing the duty of senior scientists to publish negative findingsThe arguments listed above bring every scientist to a problematic situation.

Young scientists seem to have to choose between their careers and ethical behavior. Supervisors of research groups must give good advice, otherwise they leave the young scientists alone with their decisions.

However, there are no widely accepted and widely known procedures to handle negative outcomes that are not used for publication (see some suggestions below). Therefore, currently, the best advice is:

Do not publish negative results as a young scientist.

Leave it to the senior scientists who already have a successful career and can afford to publish negative findings for the sake of good science!

It is essential to note that I do *not* suggest selective reporting.

Selective reporting is a special case of reporting bias, resulting in the incomplete publication of analyses performed in a study, leading to the over- or underestimation of treatment effects or harms.

Selective reporting is scientific misconduct. In contrast, I advise young scientists not to waste their time, grant money, and energy on studies with negative results.

In the meantime, let us work on better procedures and rules to avoid “scientific waste” and substantially reduce unnecessary repeats of the same unpublished experiments.

How do you behave ethically and handle negative results correctly?

My advice to leave the publication of negative results to the senior scientists (who already have a successful career) comes with a price. We accept the publication bias for the sake of young scientist’s careers. This promotes the reproducibility crisis.

Senior scientists cannot be trusted to take over their responsibility to publish negative results because they are not rewarded. Even worse, they may get punished by their institutions and funding agencies, which are strongly focused on scientific output, mostly quantified in impact factors, citations, and grant money.

In other words,  the academic community gets what it incentivizes. There are many ideas to improve the current science system (for example, Begley & Ioannidis, 2015), but most initiatives are still at an embryonic stage.

To my knowledge, there are no internationally accepted rules on handling negative results. A non-representative survey among 10 young professors revealed that only one had a rough idea of where to store unpublished data that may interest other scientists. Below, you will find three suggestions on how to handle the problem of negative results at the level of the scientific community:

Three strategies to improve science

The scientific community, led by international organizations like the World Health Organization and the International Committee of Medical Journal Editors, scientific societies, and national institutes of health, should emphasize the importance of negative results in fostering actual scientific progress.

There should be international guidelines for the publication of negative results, emphasizing their value and importance in scientific research. Here are three strategies to improve the current practice in the near future:

1. Registration of all studies *before* they start

Registration of clinical trials is a widely recognized tool for facilitating complete public reporting (Zarin & Tse, 2008, Williams et al, 2010) and to counter the conflicts of interests, for example, of pharmaceutical companies. 

Registration of any type of study (pre-clinical, observational, etc.) would dramatically increase the administrative load for researchers, institutions, and funders. However, it seems to be a necessary step to cure the entire research community of positive-results bias.

For clinical trials, there is a list of international registries in the Cochrane handbook. Some of these registries already allow the registration of non-clinical studies (observational, pre-clinical, etc.). 

For social sciences and other types of research, the Open Science Framework (https://osf.io) offers the possibility to pre-register studies.

2. Saving inconclusive data in publicly accessible repositories to make them available to other scientists

Since publishing inconclusive data may be tedious and considered “a waste of resources” (see above), it should be very easy and a standard procedure to save the data in publicly accessible repositories. To guarantee independence, the international scientific community should finance these through scientific societies and/or national and international funders.

Some repositories already exist – see here a list of repositories on the Nature website. Another example is arxiv.org,  a repository for documents and papers rather than data itself, including unaccepted or unsubmitted manuscripts, PowerPoint presentations, etc. Unfortunately, repositories are often costly, and researchers are not incentivized to use them.

3. Funders must oblige scientists to pre-register their studies and to make all data available (for example, in a publicly accessible repository or in a journal of the funder)

Finally, funders should take responsibility for providing a standardized procedure to pre-register all funded studies and oblige the researchers to publish all their negative findings or deposit them in a public repository. Some funders have already started their own academic journals to publish these negative findings.

A possible incentive may be to freeze the last tranche of the funds (e.g., 25%) until data are made publicly available. Without a doubt, there must be very flexible regulations because some studies are published many years after the funding period has finished.

Final advice

The final advice is simple:

As a young researcher:

  • Do *not* publish negative results. Leave it to the senior scientists!

As a senior scientist:

  • Take responsibility and publish negative findings for the sake of science, even with a lower impact factor, but do not force the young scientists to do this.

All scientists:

  • Make negative findings known in reviews (which can be published with a high impact factor) and scientific talks.
  • Push your institution, your scientific society, and/or your funders to provide a public repository for negative results and to make it easy to make negative results publicly available without the hassle of a full publication.
  • Save your negative results or inconclusive data in a public repository to make them available.
  • Help to improve the system to incentivize the publication of negative findings and replication studies – for example, talk to funders or higher education and health politicians – many listen to scientists to improve the current procedures.

Please comment below if you have any other ideas to improve the current system.

Frequently asked questions (FAQ)

How do I get negative results published?

There is no magic bullet. In my personal opinion, you have to find a journal with the best fitting scope and target audience for your study. You have to convince the editor that the journal’s readers will profit from learning about these negative results. During the review process, you have to convince the editor and the reviewers that this article will prevent other scientists from perfoming useless experiments.

Are there journals that specifically publish negative research results?

Yes, some of the most notable journals that publish negative research results include:

      Acknowledgments

      Thanks to Noémie Aubert Bonn, who provided the information about registries and repositories and helpful comments. I have used AI systems, including Grammarly, Google Bard, and ChatGPT, to enhance the English and comprehensiveness of this article. This post may contain affiliate links, meaning I get a small commission if you decide to purchase through my link. Thus, you support smartsciencecareer at no cost to you!

      Recommended reading

      The following articles may also interest you:

       

      What to do with your PhD?

      Get clarity about your career! Find your dream job in science!

      START HERE

      Career in academia?

      Learn the rules and become a professor!

      START HERE