Bibliometrics – an important tool in research evaluation

Bibliometrics uses mathematical and statistical methods to analyse and measure the output of scientific publications. Modern bibliometrics has been largely inspired by Derek de Solla Price and the seminal work was carried out by him in the middle of the last century. In the book “Little Science-Big Science” [1], published in 1963, he analysed research communication and presented a number of quantitative evaluation techniques. DeSolla Price was the first to examine the increasing trend of collaborations among chemistry researchers by using bibliometrics.

Since then bibliometrics has developed into a research field in its own right and given rise to a community of specialised experts, so-called bibliometricians. In addition, bibliometrics is used as a methodology in many other fields of science, first and foremost to map the publication pattern in different disciplines. In economics and sociology the main interest has been for cognitive purposes, that is, studying researchers’ publication behaviour.

In the last decade bibliometrics has gained increasing importance in science policy and management. It is first and foremost in the domain of research evaluation where it plays a prominent role. The development of performance indicators to respond to science policy questions has been the most common application. Indicators used for this purpose include: productivity analyses measuring the output and volume share of a specific actor e.g. a country’s world share of publications or citations; research impact analysis using citations, and relational indicators studying the heterogeneity/homogeneity or collaboration patterns between different actors.

Mapping and visualising research collaboration – a key area for science analysis and bibliometrics

Studying research collaborations has evolved into a major focus of bibliometrics and receives increasing attention from policy-makers and more general users. Modern research is regarded as increasingly complex and specialised, making it impossible for an individual researcher to master all the knowledge and technical skills needed. In a collaboration, different skills complement each other and this complementarity is hoped to stimulate knowledge sharing and the generation of innovation and new ideas. As a result, collaborative research activities do not only enable the pooling and sharing of resources for enhanced efficiency but are also linked to the quality of the research outcome.

This growing interest in research collaborations is also reflected in research funding programmes. Grants awarded by many different funding institutions and for many different disciplines often seek to encourage – and at times require as a condition – collaborations between different countries, research fields or institutions. Being able to map and analyse research networks and collaboration has therefore evolved into a key issue for the design and assessment of research policies and related funding programmes.

Bibliometrics can make an important contribution in this regard. Two or more researchers appearing as co-authors on a research publication is considered to be an indication of a collaboration. Using co-publication as unit of analysis bibliometrical studies have for example empirically demonstrated that researchers collaborate more than ever. Moreover, the share of publications with multiple institutions represented on them grew from 40% to 61% between 1988 and 2005 [2].

A more detailed example: research collaboration in Europe

In our research we use bibliometrics to study international collaboration patterns from an evolutionary point of view. Using co-authorship we explore the collaboration of researchers in Europe with those in other European countries (intra-European) and with partners outside the EU (extra-European). We examine how collaboration patterns have evolved over time against the backdrop of research policy efforts to increase European collaborations and to create a European research area (ERA).

This use of bibliometrics has yielded some important insights: We find that researchers in smaller countries co-author more with other European countries than bigger countries, while the co-authorship rate with extra-European countries is not dependent on the country’s size. We also find significant differences in collaboration patterns across fields. Engineering, Computing & Technology is the field with the highest level of national publications and Physical, Chemical & Earth Sciences the field with the highest level of both intra-EU and extra-EU collaborations [3].

We also use bibliometrical tools in the form of citations to study the visibility of different types of collaborations. Here we find that extra-European collaboration achieves higher visibility. Related co-publications have on average the highest citation rates intra-European and national co-authorships. More specifically, co-authorship with researchers in the U.S. receives the highest citation rates and thus highest visibility.

The results plausibly suggest that opening up European funding programmes, such as the EU Framework Programmes, to extra-European countries may be beneficial in terms of research impact and visibility. Today there are limited incentives for extra-European countries to join European collaboration networks because their involvement is typically not eligible for funding.

Some caveats for the use of bibliometics

As these examples show, bibliometrics provide important empirical facts to support evidence-based research policies. However, an exclusive reliance on a narrow set of bibliometric indicators has raised concerns among bibliometricians. For example, citation based metrics are often employed to rank and assess individual research papers, scientist and research institutions. In certain areas and fields these tools are directly linked to funding allocations.

With regard to citation metrics, it is important to remember that references are included in articles with different purposes in mind, for example to provide background reading, to confirm or support claims and hypothesis, or to point out the interdisciplinarity of the work. References do not necessary need to have a positive meaning but can also be included in order to criticise or point out mistakes in earlier work. Also self-citations, a reference to an author’s own work, should be taken into account when using citations as a quality measure.

n addition, when comparing countries’ publication rates it is important to remember that language can be an influential factor. Publications in journals using English as language are more likely to receive citations since the main source for bibliometric analyses, the Science Citation Index (SCI), has a bias towards English written journals.

Bibliometrically-supported comparisons between fields also need to be approached with caution. Differences here may be due to different publication strategies. Mathematicians and other theoreticians tend to publish fewer papers than researchers in experimentally-intensive fields such as life sciences. Publications in the field of physics do on average involve more authors than articles in the field of chemistry. Social scientists have on average a lower publication rate than natural scientist since the way of communicating research results often involves other means such as working papers, books etc.

Finally, with regard to mapping collaborations it is important to keep in mind that many collaborations do not result in co-published articles but may involve the sharing of research infrastructure, exchange of material or samples or some kind of informal collaboration which involve knowledge stimulation.

Bibliometricians recognise these challenges and a number of normalisation methods exist trying to minimise theses differences. Nevertheless bibliometrics can never fully replace qualitative methods of analysing research output and collaboration. Bibliometrics should rather be viewed as an important element of a more comprehensive package of tools for analysis. It provides a good overview using large scale data and can give indications on the development and productivity in a field. To be able to understand the underlying factors and the cognitive behaviour of researchers qualitative methods are needed to complement bibliometrical analysis.

[1] Derek J. de Solla Price (1963). Little Science, Big Science. New York: Columbia University Press.

[2] NSF S&E Indicators 2008.

[3] Mattsson P, Laget P, Nilsson A, Sundberg CJ. (2008). Intra-EU vs. extra-EU scientific co-publication patterns in EU. Scientometrics, 2008, Vol. 75, No. 3, 555-574.