Measuring research impact

Ways of measuring impact in terms of citations have existed for a long time, but new forms of measurement are being developed all the time. Citation-based impact figures are used on researchers' CVs, in various funding allocation to universities and institutions, and in the annually published worldwide university rankings.

Altmetrics, measuring attention or impact in channels other than scientific publications, is a phenomenon that has emerged in recent years. Exactly how altmetrics should be used is still a matter of debate.

Another way of measuring impact is to see whether scientific publications are cited in policies and guidelines. Overton is a database that collects policy documents and links them to research.

T o t h e Overton database

The varied function of citations

Citation-based metrics such as the Journal Impact Factor (JIF) and the h-index are often used to measure the impact of research. Sometimes this is equated with the quality of the research. One reason why various forms of bibliometric measures have taken on an increasingly important role in the assessment of research and researchers is the increasing volume of publications. Counting publications and citations is simply perceived as more efficient than to qualitatively assess the contribution of the specific publications to the field of research.

Relying solely on citation measures, however, implies a number of well-known problems. It is useful to be aware of these issues when assessing your own research, or that of others. In addition, the problems are one reason why it is problematic to compare different subject areas based on citation measures.

Citation measures are extracted from databases that provide better coverage in some subject areas than others. Subject areas where publication rests largely on genres other than journal articles, and in languages other than English, are generally poorly represented in bibliometric studies.
Citation practices differ across subject areas. Texts are expected to refer to other texts – but to different extents. The function of citation may also differ. For some subjects, citation may indicate high quality of the cited text, because a low quality text would not be cited. In other fields, citation may occur because the cited text is questioned or refuted.

As a consequence, citations can be said to measure impact and visibility to some extent, and to varying degrees depending on subject areas; however, they do not (or do not necessarily) measure quality.

There is an international debate on citation measures and several major stakeholders have argued that such measures should be used with caution (e.g. in the Declaration on Research Assessment (DORA) signed by Forte, Formas, RJ and other organisations). At the same time, it is an extensive undertaking both in terms of information legwork and attitudinal work, to change the perceptions among researchers, experts and academic leaders regarding the importance of measures such as the JIF. This means these measures are likely to be of continued great importance in many sections of the scientific system.

What you can do

As an individual, you can be mindful of how the measures are developed and used. Compare the numbers retrieved from different databases and consider why the numbers differ. Present plainly the source and date when providing any measure. Familiarise yourself with the limitations of various measures, so you can both relate them in a way that causes no disadvantage to you, and act responsibly when using measures to judge others. Not least, consider the publishing and citation practices of the subject area in question.

Level of publication

The number of citations a specific publication has received (how many times it appears in the reference lists of other scientific publications) is not just potentially satisfying data for the author of the publication but will also inform you about the publication's impact.

Citation databases, such as Web of Science and Scopus, systematically collect and generate statistics based on citations. Google Scholar also identifies citations algorithmically. Citations are measured in different ways, and links are established between the cited and the citing publications. An increasing number of other databases also include citations, but not always as exhaustively.

In recent years, alternative measures of impact and visibility have seen budding use, both within specific databases and on journal websites. For example, the publisher PLoS reports number of page views directly on the article pages, as well as number of downloads and shares of the article made from the page. Read more about this under Altmetrics.

Journal level

Det The best-known and most widely used bibliometric measure at journal level is the Journal Impact Factor (JIF), which is calculated from the data of the Web of Science database. The JIF is often used as an indication of the value of a specific journal, and has come to rub off increasingly on individual articles and their authors, to the point where a researcher's output is sometimes valued on the basis of the JIF of the journals in which an article is published, rather than the value or impact of the individual article.

For the same reasons as those above, JIF should not be considered a measure of quality – not for the journal and certainly not for the individual article, but as a measure of impact or visibility.

This is JIF

JIF is calculated on the basis of the average number of citations to the articles in a journal within two years of the year of publication (known as the citation window).

Formula for the journal impact factor for 2022, based on citations from 2020–2021.

For some subject areas, a two-year citation window is very short, as citations from an article in the given field are often spread out over a much longer period of time. Information on Web of Science journals and how they rank against each other is available in the InCites Journal Citation Reports database subset. Journals within the humanities are not included in the JIF.

This is CiteScore

The Scopus database equivalent to JIF is called CiteScore. It is basically very similar, but uses a four-year citation window instead of a two-year one. Journals can be browsed via the Sources tab in Scopus.

Criticism of JIF and CiteScore

One objection that has been made against JIF and CiteScore is that it is not possible to compare the indicator across different subject areas. What is a very high JIF within one field is a very ordinary JIF in another, due to different citation practices. For example, top-ranked journals in mathematics have a JIF around 4, while a comparable ranking means a JIF over 30 within cell biology (Leiden Manifesto and InCite JCR). In response to this, field-normalised citation indicators (based on what is an expected normal value in a given field) have been developed, such as the Journal Citation Indicator (JCI) for Web of Science and the Source Normalised Impact per Paper (SNIP) for Scopus. These indicators allow comparison of journals across subjects, which should still be done with caution.

Another way of comparing journals across subjects is to look at the percentile within which they fall in their subject category (e.g. in the top 10% or top 25%). There are a number of other indicators that have also been proposed in recent years, such as the Eigenfactor Score and the SCImago Journal Rank, both of which are based on a logic similar to that of Google's PageRank algorithm.

JIF and CiteScore are two simple metrics that entice people to rank journals, and invite attempts to get published in journals that are highly ranked. However, these metrics have been criticized for, among other things, the following issues:

the focus is on where an article is published (in what journal), rather than the quality of the article
a high JIF/CiteScore may be due to a small number of articles receiving a very high number of citations while the majority of articles in the journal have a much lower number, or no citations at all
the citation window for some topics is too short to capture the bulk of the citations
journals can influence the results in different ways (e.g. by systematically encouraging citations to articles in their own journal or by publishing articles that are expected to receive many citations early in the year to ensure that as many citations as possible fall within the citation window)

(Sugimoto & Larivière, 2018, p. 94)

In addition, there are several initiatives to discourage the use of journal-level metrics as a measure of individual article quality, impact or prestige, see for example the DORA Declaration.

For obvious reasons, a similar citation-based measure for journals does not exist for books and book chapters, although Scopus in particular does measure citations at the document level.

Personal bibliometrics (H-index)

The H-index is a measure developed for comparison of researchers based on the intersection of values for productivity (number of publications) and attention (number of citations).

To obtain an H-index, an author's publications are listed in order, from the one with the highest number of citations to the one with the lowest number (or no citations at all). The h-index is the figure where the number of citations for a publication is equal to or higher than the publication's position in the list. For example, if you have authored ten publications with at least ten citations to each, but fewer than eleven citations to the eleventh publication in the list, your h-index is 10.

A table shows an H-index of 10; ten publications have at least ten citations.

The h-index is derived from a database that records citations. Common databases to use are Web of Science, Scopus or the Google Scholar search engine. These databases differ in the control over what they include, as well as regarding what genres are included. This makes it inappropriate to compare h-indexes produced through different databases. For example, Google Scholar will almost always produce a higher h-index than the other two databases mentioned above, and the publications from which citations are drawn in Google Scholar may include, for example, student papers.

It is also important to not compare h-indexes between different subject areas, as publishing and citation practices can differ greatly. What is considered a high h-index in one field is not necessarily as impressive in another. There have been proposals to develop field-normalised h-indexes, which relate indices to the averages within a given subject, but this has not caught on.

If you are expected to provide your own h-index, it may be useful to create a profile in the database from which the h-index is to be retrieved, and ensure that all your publications (which are indexed in the database) are linked to your profile, and that the list excludes all publications that are not yours.

In some fields, such as those where national publishing is common, Web of Science and Scopus may contain very few publications out of a researcher's total production. In this case, there may be good reason to use the h-index from Google Scholar alongside the h-index from, for example, Web of Science, and for arguing why publications and citations in Google Scholar tell you more.

Benefits of h-index

It measures citations to the author's publications rather than to the journal in which an article is published. Citations to the journal do not tell us anything about the impact of the individual article.
The measure is less sensitive to individual, very highly cited publications.
It does not favour authors who have published very widely but have not received any citations.

Disadvantages of h-index

It does not take into account the extent to which an author has contributed to a co-authored publication, but instead all authors are counted as equals.
It is a measure that favours more senior researchers as the number of publications limits the h-index that can be attained.

Altmetrics

The rise of electronic publishing and the use of unique identifiers such as DOI has opened up opportunities to measure impact in a broader and alternative way, beyond the classic citation-based analyses. These practices are often referred to as altmetrics or alternative metrics. Rather than being limited to regular citations in other scholarly works, they also take into account mentions in places such as policy documents, news, social networks and various kinds of websites.

One possible consequence of this is that impact is generally seen more quickly with altmetrics, as in some cases the citation is made directly without a publication citing the work having to be written and then undergo a review and publishing process.

There are two main providers of altmetrics, Altmetric and the publisher Elsevier's PlumX tool (some of the PlumX metrics are available through Scopus). To find Altmetric's figures for a specific publication, download their Altmetric Bookmarklet plug-in.