Which bibliometric data are relevant for a research career?

For any scientist pursuing a career in academia, bibliographic metrics such as impact factors or citations are essential for measuring the impact and relevance of their research. But which bibliographic metrics are relevant, and which are misleading?

Why are bibliographic metrics needed?

Science administrators need quantitative metrics

Quantitative metrics are needed by science administrators who have to compare the research performance of applicants for grant proposals or academic positions in a quantitative manner. 

The principal idea is well-intentioned to avoid biases in selection procedures. However, all quantitative metrics are just proxies to measure scientific quality and are subject to intensive debates.

As a consequence, you must learn how to increase the impact factor of your publications and get more citations.

Administrators love comprehensive coverage and data availability

It is bad news for you and your career that administrators love comprehensive coverage of bibliometric parameters because you have to satisfy them when submitting applications for grant proposals and positions.

In addition, most academic research institutions are obliged to document their excellence in research comprehensively. Thus, it is your unpleasant duty to document all your publications and deliver the bibliographic data to your institution.

Why is this necessary?

Bibliometric analysis and bibliometric methods are critical for evaluators to statistically analyze and compare the research output of individual scientists, research groups, and entire research organizations.

Similarly, research evaluation plays a significant role in informing national and international research policy decisions. Funding organizations and policymakers must evaluate the national research output of various research institutions and compare the international research output across different countries or, for example,  European research versus global research.

Bibliometric indicators are essential to quantify scientific quality

To accurately assess the impact and quality of scientific research, achieving comprehensive coverage of the research literature is essential for research managers. 

This includes incorporating research articles, research papers, conference proceedings, book chapters, and other forms of scholarly literature into the analysis. Information science and library science are two major fields where a bibliometric approach is crucial in the evaluation of research.

Citation tracking tools like Google Scholar, Web of Science Core Collection, and Scopus aim to provide comprehensive coverage of the scientific literature across multiple disciplines, including the natural sciences, social sciences, health sciences, and computer science. 

By capturing and analyzing citation data from various sources, researchers and evaluators can better assess the impact and quality of research output quantitatively.

The quantitative assessment is wanted to avoid biases in selection procedures – because the quality of scientific output is very difficult to evaluate without continuous struggle and heavy debate.

In addition to traditional citation data, researchers in information science and library science also utilize data sets, white papers, and research findings to evaluate research quality. 

During a research assessment, they often employ a mix of quantitative data and qualitative data to gain a more comprehensive understanding of research impact and performance.

Evaluators often use bibliometric data, including citation indicators, to compare the impact and quality of research across various scientific disciplines. Naturally, there is a heavy debate about whether measuring scientific quality in highly diverse fields of research is reliable – or fair.

For example, in the United States, the National Science Foundation (NSF) is a major source of future funding for scientific research. The NSF utilizes different bibliometric metrics to evaluate grant proposals and assess the impact of funded research.

To reduce biases, the NSF encourages using multiple metrics, including traditional metrics like citations and h-index, and alternative metrics like altmetrics.

In the United Kingdom, the Research Excellence Framework (REF) is a critical assessment exercise used to evaluate the quality of research conducted at higher education institutions. 

The REF considers bibliometric metrics such as citation counts, h-index, and altmetrics, along with peer review and expert judgment, to allocate future funding. This analysis includes examining citation data, citation measures, and alternative metrics to measure the impact of research papers, book chapters, and other scholarly literature.

This is important for your career because you must adapt to this development. These methods are already standard by many funding agencies and academic institutions to allocate resources and recognize outstanding research contributions.

Using bibliometric data responsibly

The choice of data source for bibliometric metrics can significantly impact the results obtained. Different databases like Web of Science, Scopus, and Google Scholar cover different sets of publications, leading to potential variations in citation counts, h-index, and other metrics.

Researchers and evaluators must be aware of these differences when comparing bibliometric research data across various levels of analysis, such as individual researchers, research groups, or institutions.

The use of bibliometric metrics has led to the development of guidelines and declarations, such as the San Francisco Declaration and the Leiden Manifesto, which emphasize the importance of using responsible metrics.

These declarations promote transparency, accuracy, and ethical use of bibliometric metrics in evaluating research output. They encourage the consideration of multiple research metrics, including traditional metrics and alternative metrics, in assessing the quality and impact of scientific research.

The most relevant metrics for your career in science

There is a long list of bibliometric indicators. Deciding which metrics are important for your career in science is difficult. However, my very personal opinion is that only four parameters are worth considering for different reasons: Impact factors, citations, h-index, and altmetrics.

This pragmatic approach is based on many years of personal experience as a member of multiple selection committees for grant proposals and academic positions (PhD, postdoc, tenure-track and experienced professors).

Impact Factor

What Are Impact Factors?

The impact factor (IF) or journal impact factor (JIF) of an academic journal is an index calculated by Clarivate that reflects the yearly mean number of citations of articles published in the last two years in a given journal as indexed by Clarivate’s Web of Science. 

It is frequently used as a proxy for the quality of a journal within a specific research field.

High impact factors are generally considered to indicate that the journal is publishing high-quality research that is being widely cited by other researchers. Thus, publishing in academic journals with higher impact factor values is considered more prestigious. It makes your job or grant application more attractive.

The history of the impact factor and the debates about its usefulness are complex. Read more in this Wikipedia article. Below I provide a short summary of the criticism.

Criticism of using citation numbers to quantify scientific quality?

Impact factors are still frequently used by administrators and reviewers of universities and funding bodies, for example, to decide on promotion and research proposals. However, impact factors faced criticism from the scientific community. Some of the main criticism of impact factors include:

Impact Factors are Journal-Based, Not Article-Based:

Impact factors are calculated at the journal level, which means that articles published in highly ranked journals may receive more citations simply because of the journal’s reputation rather than the quality of the individual article.

Inapplicability of impact factor to individuals and between-discipline differences

The use of impact factors and citation analysis to evaluate published research has been criticized as being affected by field-specific factors, making comparisons across disciplines and even within different fields of one domain invalid. 

The percentage of citations occurring in the first two years after the publication date also varies greatly among disciplines, making impact factors unsuitable for comparing journals across different fields.

Questionable editorial policies that affect the impact factor

Some journals use questionable editorial policies to inflate their impact factor. One tactic is to publish a higher percentage of review articles, which are more likely to be cited. Another is to limit the number of “citable items” by not publishing articles that are unlikely to be cited.

Some journals also publish highly cited papers early in the year to give them more time to accumulate citations. Journals can also use overt measures to game the system, such as coercive citation, where journal editors force authors to add extraneous citations to inflate the journal’s impact factor.

Why are impact factors important for your career in science?

Impact factors are crucial in choosing suitable journals to publish your research. A higher impact factor indicates a journal’s quality and prestige within a field, increasing the visibility and credibility of your work. 

By targeting high-impact journals, you enhance your chances of being cited, securing funding, and advancing your career. However, also consider the journal’s scope, relevance, and publication turnaround time to maximize your research’s potential impact.

Citations

What is a citation?

A citation is simply a reference to a source that supports a statement. For example, I may state in a review: “Targeting the phagocytic ability of macrophages may have therapeutic potential for the treatment of spinal cord injury.” This statement is followed by a reference supporting the concept – for example, an excellent review by one of my former PhD students: Van Broeckhoven et al., 2021.

You will find the complete reference in the reference list at the end of the review. A citation can be given in many formats, such as in the form of a number (2) or the name of the first author and the year, such as Van Broeckhoven et al., 2021.

How are citations counted?

When a researcher cites a paper as described above, they reference the original research paper in their bibliography. Citation tracking tools, such as Google Scholar, Web of Science, or Scopus, collect this data and also use it to create citation metrics, such as impact factors and h-indexes.

Citations directly measure the impact and influence of a researcher’s work. They indicate how often a researcher’s work has been referenced and built upon by others. In other words, high citation counts represent a researcher’s standing.

To my knowledge, citation tracking tools like Google Scholar, Web of Science, and Scopus Cite a variety of sources and generally do not distinguish between a non-peer-reviewed and a peer-reviewed article when calculating citation metrics.

The Citation tracking tools also track those conference papers or conference proceedings that are saved in their databases. Still, they are not tracked like journal articles, so counting citations to a conference paper can be harder.

Determining the citation rate for a book or book chapter can be more challenging because citation counts for books are a relatively recent endeavor and often less accurate than those for journal articles.

How to count your citations with Google Scholar Citations?

Creating a Google Scholar Citations profile gives you a tool to assess research performance of your own work and that of others. You can track your article citations, calculate citation metrics such as your h-index, establish a public profile to showcase and receive updates on new research citations and relevant articles in your field.

Using Google Scholar Citations is particularly useful for researchers who want to understand the impact of their work and follow trends in their research area. 

Google Scholar data may indicate which new research lines are more promising for getting funding and whether the research may be attractive for collaborative research in a consortium.

Criticism of using citation numbers to quantify scientific quality

The major criticism of using citations to measure scientific quality is that it only provides a narrow view of a publication’s impact and influence. The number of times a paper receives a citation can be influenced by various factors, such as the size or visibility of the field.

One of the main reasons bibliometric information may not accurately reflect a researcher’s impact is the difference in the period over which citations accrue. 

In some fields, research papers may accumulate citations rapidly within the first few years of publication, while in other areas, citations may accumulate more slowly over a longer period.

Citations may not capture the full impact of a publication, such as its impact on public policy or society as a whole. Furthermore, the citation process can be manipulated (see below) or biased, as researchers may cite papers from their colleagues or prestigious institutions rather than the most relevant or innovative work. 

Despite these limitations, citations remain a widely used metric for evaluating scientific research.

Citation Patterns and Impact

Due to the fast development of new artificial intelligence (AI) tools, I dare to predict that citation analysis will become as professional as search engine optimization (SEO). 

I expect dramatic changes in the strategic analysis of citation patterns and the rise of powerful AI research tools for citation analysis. In other words, getting proficient in obtaining more citations for a career in academia will become necessary.

In recent years, bibliometric studies have entered a second phase characterized by advanced analysis tools that aim to provide a more nuanced understanding of citation patterns and research impact. These tools use machine learning and AI techniques to explore complex citation networks and identify emerging research topics and trends.

Citation patterns have already gained attention as publishers, editors, researchers, and evaluators attempt to understand the complex nature of how scientific literature is cited and how citations can be manipulated.

These patterns can reveal the citation impact of a paper, showing the extent to which it has influenced subsequent research. Additionally, by analyzing citation patterns, researchers can better understand the structure of citation networks and how these networks change over time.

For your scientific career, this knowledge can be valuable for evaluating and assessing research impact across various scientific fields and research areas and for strategies leading to a higher value in bibliometric metrics.

Bibliometric mapping

Bibliometric mapping is a technique used in bibliometric and scientometric research to visualize the relationships and patterns within a research domain or field. 

This method involves analyzing and representing large volumes of bibliometric data (e.g., publication data, citation data, and keywords).

The resulting data visualizations, often in the form of graphs or networks, help researchers, policymakers, and administrators gain insights into the structure, dynamics, and trends within a specific research area.

Why are citations important for your career in science?

Citations are essential for your career as they measure specifically your personal impact and visibility in your field. High citation counts indicate that your work is widely recognized and influential. This is the basis for getting funding, job offers, promotions, and collaborations.

Additionally, your academic employers will probably use citation metrics to evaluate your job performance.

Finally, when your career is more advanced, you may become proficient in recognizing citation patterns and understanding bibliographic mapping to optimize your publication strategy.

What is the difference between Journal citation report, citation count, and impact factors

The Journal Citation Reports (JCR), citation count, and Impact Factors are connected but serve different purposes.

Journal Citation Reports (JCR)

Journal Citation Reports is a database that provides various metrics (including the Impact Factor) published annually by Clarivate Analytics (previously part of Thomson Reuters). 

It provides citation and publication data of academic journals across various disciplines. JCR allows users to evaluate and compare journals using citation data from various scholarly and technical journals.

Citation count

As described above, citation count is a metric to quantify the impact of an individual article (and its authors) on the research community. It is the total number of times a specific scientist’s published work (e.g., an article, book, or conference paper) has been cited by other published works. These data are collected by citation tracking tools, such as Google Scholar, Web of Science, or Scopus.

Impact Factors

As described above, the impact factor of an academic journal is a metric to quantify the impact of a specific scholarly journal on the scientific community. 

It reflects the yearly mean number of citations of articles published in the last two years in a given journal. The impact factor is one of the key metrics provided by the JCR database.

What is the relation between JCR, citation count, and Impact Factors

The Journal Citation Reports (JCR) database uses citation counts to calculate various metrics, including Impact Factors, to evaluate and compare the performance and influence of academic journals across disciplines.

The h-index

What is the h-index?

The h-index is a metric used to quantify a researcher’s performance (in contrast to impact factors or citations used to quantify the quality of a journal or a single publication). 

The h-index is sometimes called the Hirsch index or Hirsch number. It measures the number of papers a researcher has published that have been cited at least that many times.

For example, an h-index of 20 means that a researcher has published at least 20 papers that have been cited at least 20 times each. 

Interestingly, the h-index correlates well with academic success indicators such as winning the Nobel Prize, getting research fellowships, and holding positions at top universities.

Criticism of using the h-index to quantify the performance of a researcher?

The h-index is a popular bibliometric measure used to evaluate the performance of individual researchers. However, there are several criticisms of the h-index as a sole indicator of a researcher’s performance:

Career stage and publication history influence the h-index.

The h-factor Is heavily influenced by the researcher’s career stage and publication history. Early-career researchers commonly have a lower h-index despite their promising research activities, while established researchers may have a higher h-index due to their longevity in the field.

The h-index does not consider the quality or impact of research publications.

The h-index does not consider the quality of scholarly journals, as it treats all publications equally irrespective of their place of publication or their influence level in their respective fields. Self-citations can artificially inflate the h-index. Thus, doing good research is not necessarily reflected in a good h-index.

Collaboration is not reflected in the h-index.

The h-index does not reflect collaborative work, as researchers who collaborate extensively may have a lower h-index despite their contributions to significant research projects.

No account for disciplinary differences in citation practices.

The h-index does not account for the disciplinary differences in citation practices, as some fields may have higher citation rates than others.

Therefore, while the h-index can be a valuable metric for evaluating researchers’ performance, it should not be the sole indicator and should be used in conjunction with other measures, such as the quality and number of research publications, awards, grants, and peer recognition.

Why is the h-index important for your career in science?

The h-index is a critical metric for evaluating the impact of a scientist’s research output over a more extended period. In other words, the h-index is used to assess more established researchers. 

Thus, the h-index is usually ignored when selecting postdocs or tenure-track professors because they are still at the beginning of their careers.

In contrast, when selecting associate or full professors or senior group leaders, the h-index is regularly used to compare candidates. Thus, it is essential to systematically work on increasing your citation count and h-index.

Altmetrics

Social media sites have a significant impact on how scientific research is disseminated and discussed. Researchers are increasingly sharing their work on platforms like Twitter, Instagram, Facebook, and LinkedIn, leading to a more rapid spread of information and potentially a higher citation count.

As a result, some researchers argue that the presence and engagement on social media sites should be considered when evaluating the impact of a researcher’s work.

Altmetrics are a relatively new type of bibliometric metric that measures the attention and engagement that a particular article receives online, including mentions on social media, blogs, and news outlets. 

Altmetrics can provide a more immediate and nuanced view of the impact of a researcher’s work beyond traditional citations.

One major criticism is that altmetrics can be easily manipulated and are not standardized across different platforms. For example, social media mentions and shares may not always be a reliable indicator of a publication’s quality. They can be influenced by factors such as sensationalism or the author’s popularity.

Altmetrics may not capture the long-term impact of a publication, as they tend to focus on short-term attention and engagement. Despite these criticisms, altmetrics may still be a helpful supplement to traditional citation-based metrics.

Why are altmetrics important for your career in science?

I expect that altmetrics will be an essential aspect of evaluating a researcher’s excellence in the coming years. Although their validity can be heavily debated, they will be used as an additional parameter in addition to citations, impact factors, and the h-index.

As a consequence, you should consider an efficient social media strategy to be able to compete with other scientists of similar qualifications.

Are other metrics important for your career in science?

There are many other bibliographic metrics for evaluating journals and for assessing individual researchers, such as the Eigenfactor, CiteScore, SNIP, FWCI, i10 index, and many more.

However, in my experience, most scientists are not familiar with them, and they are not commonly used by selection committees for academic positions or grants. 

Thus, I suggest ignoring these other parameters – for now – and focusing on impact factors, citations, h-index, and altmetrics.

Acknowledgments

I have used AI systems, including Grammarly, Google Gemini, and ChatGPT, to enhance the English and comprehensiveness of this article. This post may contain affiliate links, meaning I get a small commission if you decide to purchase through my link. Thus, you support smartsciencecareer at no cost to you!

Recommended reading

The following articles may also interest you: 

• How To Write Faster: 19 Efficient Ways To Finish My Publication
• 10 simple strategies to increase the impact factor of your publication
• 28 Tips to Get More Citations for Your Publications 
• Should I publish negative results, or does this ruin my career in science?
• What is a substantial contribution to a paper?
• Should I aim for co-authorships on high-impact papers?
• Should I aim for multiple co-authorships to extend my publication list?
• Should I have senior authorships as a postdoc?
• I have a fake author on my paper – what should I do?
• Do I need nature or science papers for a successful career in science?
• What is the best publication strategy in science?