University rankings – what do they measure?
In Issue 9 Research Trends examined the Times Higher Education-QS World University Rankings, and we explored how the rankings of institutions in different countries have changed over the years. In this article we revisit university rankings from a country and regional perspective. Two of the most widely known world university rankings — the THE-QS World University Rankings* and the Academic Ranking of World Universities (ARWU) produced by Shanghai Jiaotong University — measure the performance of universities using a range of indicators, which are combined into an overall score that is used to determine the university’s rank. But how do different countries and regions perform for the various indicators on which their overall scores, and therefore rankings, are based? We investigate this question using data from the 2009 ranking exercises.
*In 2010 the THE-QS ranking split into two new ranking schemes: the first, produced by QS, continued with the same methodology; the second, produced by THE in Collaboration with the information company Thomson Reuters, used modified methodologies, indicators, and data sources.
The THE-QS ranking uses six indicators: academic peer review; employer review; faculty to student ratio; citations per faculty member; proportion of faculty that are international; and proportion of students that are international. An overall analysis of the average values for these indicators by country reveals an interesting pattern in the two ‘internationality’ indicators (see Figure 1): these measures tend to be higher in small, wealthy countries with a strong research base, such as Singapore, Ireland, Switzerland, and Hong Kong. The small size of these countries means that they have a relatively small domestic pool of students and researchers relative to the global pool, which as a result will tend to be strongly represented at these universities. We also see high values of these measures for countries with a global research reputation, English language culture, and strong international links (see Figure 1), such as the UK (popular with students globally, due to historic research culture and reputation), and Australia, which is a popular higher education centre in the South Asia region. Interestingly, the US, whose institutions dominate the THE-QS top 200, scores relatively low for measures of ‘internationality’; this could be attributable to a number of factors, including regional isolation, high costs, and the large pool of domestic students and researchers available to populate the universities.
Figure 1 – Average scores for ‘internationality’ indicators in the 2009 THE-QS World University rankings, selected countries.
The ARWU scheme also uses six indicators: major academic awards to alumni; major academic awards to staff; researchers in Thomson’s Highly Cited lists; publications in Nature and Science; volume of publications in Thomson’s Science Citation Index; and per capita performance (a score based on the first five indicators that is weighted by the number of staff at the university). A country-by-country breakdown of indicator scores (see Figure 2) reveals that countries with a strong English language culture perform well for the Highly Cited indicator (a measure based on data from the Web of Science database, which has a strong English language emphasis).
Figure 2 – Average scores for each indicator in the 2009 ARWU ranking, selected countries. Alumni = Weighted count of alumni winning Nobel Prizes or Fields Medals; Award = Weighted count of staff winning Nobel Prizes or Fields Medals; HiCi = Count of highly cited researchers in 21 subject categories; N&S = Count of articles published in Nature and Science in recent 5 years. PUB = Count of articles indexed in Science Citation Index-Expanded and double the count of articles indexed in Social Science Citation Index in recent year; PCP = Per Capita Performance: weighted scores of the above five indicators divided by the number of FTE academic staff. Asterisks (*) indicate countries with a notably high HiCi score.
Scaling up to regional level in the ARWU rankings, it emerges that institutions based in North America (the US and Canada) outperform institutions in other regions on average, according to the Highly Cited andNature/Science publication indicators, both of which are measures of high impact research (see Figure 3). In contrast, we see that institutions in the Asia-Pacific region perform poorly for the two indicators that measure major awards to alumni and staff.
Figure 3 – Average scores by region, for indicators in the 2009 ARWU ranking. Indicators as per caption for Figure 2.
In 2009, both the THE-QS and ARWU university ranking schemes showed substantial country-level and regional-level differences in indicator scores. This raises the question of whether the indicator scores effectively measure the quality of individual universities, or whether they are too strongly influenced by global variation in the higher education/research system to allow meaningful comparisons between institutions that are located in different geographical zones.