- Split View
-
Views
-
Cite
Cite
Elissa Z. Cameron, Angela M. White, Meeghan E. Gray, Solving the Productivity and Impact Puzzle: Do Men Outperform Women, or are Metrics Biased?, BioScience, Volume 66, Issue 3, 01 March 2016, Pages 245–252, https://doi.org/10.1093/biosci/biv173
- Share Icon Share
Abstract
The attrition of women from science with increasing career stage continues, suggesting that current strategies are unsuccessful. Research evaluation using unbiased metrics could be important for the retention of women, because other factors such as implicit bias are unlikely to quickly change. We compare the publishing patterns of men and women within the discipline of ecology and show sexual dimorphism in self-citation leading to higher h-index scores for men despite lower citations per paper, which is exacerbated by more career absences by women. However, if self-citations and non-research active years are excluded, there are no gender differences in research performance. The pattern is consistent across disciplines and may contribute to current geographic disparities in research performance, rewarding confident behavior and traditional career paths rather than research impact. Importantly, these changes would not disadvantage anyone, because self-citation does not indicate broader impact, and researchers should only be judged on their research-active career.
Women are still not progressing in science careers and become increasingly underrepresented with advancing career stage, a phenomenon known as the leaky-pipeline problem (Pell 1996). Although the early stages of the pipeline from undergraduate to postgraduate study no longer leak more women than men (Miller and Wai 2015), the ongoing nature of the problem at higher levels has received considerable attention in the scientific literature (Moss-Racusin et al. 2012, O'Brien and Hapgood 2012, Sugimoto et al. 2013), including recent high-profile editorials (McNutt 2013, Shen 2013), and more recently came to broader attention through the injudicious comments of a Nobel Laureate (e.g,. Sexism Has No Place in Science 2015). Such discussions underscore that despite the advances made, women continue to face barriers to progression in a research career. Decisions to leave academia have been largely attributed to maternity and differences in family responsibilities, either by choice or constraint (Ceci and Williams 2011, Whittington 2011, McGuire at al. 2012). However, perpetuating the idea that an academic career is incompatible with raising a family may itself contribute to the attrition of women faculty (Sax et al. 2002) and cause the dismissal of other factors that result in a loss of women from the sciences. Furthermore, if factors unrelated to maternal responsibility contribute, efforts such as creating more flexible workplaces or better tolerance of part-time work (Scandura et al. 1997, O'Brien and Hapgood 2012) may still prove ineffective. For example, lower scientific self-confidence, mediated mainly through societal influences, may contribute—particularly if expressed through different publication patterns.
Research aptitude and performance are judged on publications (Cameron et al. 2013), and metrics have become increasingly important for promotion and grants (Fischer et al. 2012). However, it has been suggested that “the careless application of metrics is likely to further reduce female participation in research” (O'Brien and Hapgood 2012). Conversely, however, the sensible use of unbiased metrics could be particularly important for improving the retention of women, because other factors (such as subconscious bias in the evaluations of female applicants; Steinpreis et al. 1999, Moss-Racusin et al. 2012, Sheltzer and Smith 2014) are unlikely to be easily or quickly addressed. However, a problem arises if metrics are not equitable but are considered unbiased in evaluations.
Several studies have shown that there are sex differences in publication patterns, with women generally publishing less than men (Addessi et al. 2012, McGuire et al. 2012, Sugimoto et al. 2013), although citations can be higher, resulting in women having higher impact per publication (Symonds et al. 2006, Goulden et al. 2011, Duch et al. 2012). Therefore, concentrating solely on publications or publication rate could contribute to women being perceived as lower achievers and thereby having lower grant success (Martin 2012), slower promotion (O'Brien and Hapgood 2012, Shen 2013), and lower retention rates (Goulden et al. 2011, O'Brien and Hapgood 2012). These differences could be due to a variety of factors, including family responsibilities and part-time employment (Whittington 2011, O'Brien and Hapgood 2012), different time expectations with higher service commitments and teaching loads (, O'Brien and Hapgood 2012), lack of role models or mentors (Damschen et al. 2005, Martin 2012), or different experiences of the scientific community leading to lower confidence (Cameron et al. 2013). Recent studies have suggested that gender biases in publication rate have disappeared (van Arensbergen et al. 2012, Pautasso 2013). But even if gender biases have been reduced, studies continue to show that women make less money, receive fewer grants that are of an average smaller amount, are hired proportionally less as faculty, and make up a minority of tenured faculty (Handelsman et al. 2005, DesRoches et al. 2010, Bedi et al. 2012, Martin 2012, West et al. 2013), suggesting that gender disparity is still a real issue in academia.
Research metrics aim to provide a level playing field to compare researchers for grant success, employment, and promotion, and they should therefore rely on the absence of systematic bias. Productivity and impact are the two most commonly used metrics of research performance. Although there has been much discussion around the effectiveness of the h-index (Hirsch 2005), particularly in relation to potential manipulation by self-citation (Purvis 2006, Engqvist and Frommen 2008, Leblond 2012, Wallace et al. 2012), it remains the most commonly used impact metric. Studies suggest that average self-citation rates are relatively constant (around 10%; Slyder et al. 2011, Leblond 2012, Wallace et al. 2012, King et al. 2015) and that strategic self-citation has only a short-term effect (Engqvist and Frommen 2008). However, these metrics assume that there is no widespread, systematic bias in research productivity measures, despite some studies showing sex-specific differences in self-citation (Hutson 2006, King et al. 2015). Indeed, some have suggested that men and women should be considered separately when evaluating research performance (Abramo et al. 2015), but few have considered that gender disparities may arise because metrics are not calculated in an equitable manner rather than because women underperform.
Here, we test for differences in publication patterns between men and women in ecology, a field dominated by women at undergraduate levels (Martin 2012).
Methods
We assembled the entire list of authors from six tier-one journals in ecology in 2011 (Ecology, Journal of Applied Ecology, Journal of Animal Ecology, Ecology Letters, Conservation Biology, Behavioral Ecology), which yielded 3350 authors. We then classified each author by gender when possible by confirming the author's gender identity on websites and used the database Scopus to obtain all publication metrics relating to their whole publishing career in any journal for those authors that could be disambiguated from other authors with similar names. In an attempt to control for historical influences, we only assessed authors who began publishing in 1994 or later, because the proportion of women PhD graduates reached over 40% in 1993 (National Science Foundation 2013) and the bachelor-to-PhD pipeline stopped leaking more women than men during the 1990s (Miller and Wai 2015), suggesting that many of the earlier explicit biases should have been reduced by 1994. In order to calculate a current publication rate based on the last 5 years, we further limited the main data set to those that first published in 2008 or before, resulting in a data set of authors that first began publishing in any journal sometime in the 15 years between 1994 and 2008. Because we are interested in these established researchers, we further limited the data set to those we considered currently research active: having published at least five papers in total and at least three in the last 5 years. In order to determine each researcher's current geographic region of affiliation for current research activities, we used the address assigned by Scopus, or where there were multiple addresses or some ambiguity, we used the address of their most recent publication. This yielded a total of 1512 individual authors used in the analysis, and a summary of authors contributing to the analysis is shown in figure 1. We also conducted a multidisciplinary analysis, for which we used authors publishing in November or December 2011 in Science or Nature, in sequential order using the same constraints as we described above until we had a total of 140 males and 140 females. Data were analyzed by t-tests with Bonferroni corrections, and data were transformed to a normal distribution where appropriate. Analyses that calculated coefficient estimates for gender and career stage were also calculated (supplemental material S1), but the conclusions were identical, so we present the simpler analysis. All analyses were conducted using R.
Productivity
Men publish more papers than women, both in terms of total numbers of publications throughout their career (table 1, figure 2a) and during the last 5 years (table 1), such that historical factors alone cannot explain the difference. Consequently, men have a higher yearly rate of publication (table 1), indicating that the productivity pattern is not disappearing in ecology, as suggested by some authors (Pautasso 2013), but confirms it is established early in the career (Symonds et al. 2006). Resource distribution drives publication patterns (Duch et al. 2012), such that early differences in publication rate could determine access to research resources, compounding sexual dimorphism over time (the “Matthew effect”; Merton 1968). However, we also tested whether career absences, possibly due to maternity and often coinciding with early career (Adamo 2013) or time and allocation differences (Link et al. 2008), influence publication rates. As expected, women were more likely to have complete years without publications (table 1, figure 2b). Although the absence of publishing during an entire year is a crude measure of career absence, a career gap of even a single year has previously been shown to influence citation impacts (Ioannidis et al. 2014). If we consider publication rate per publishing active year (i.e., at least one publication), the difference between men and women was reduced, but the difference was still significant (table 1). Many employers and granting agencies attempt to control for career absences to address gender inequities. Our analysis suggests that periods of research inactivity are important for explaining lower rates of productivity and therefore should be controlled for in publication metric calculation but are insufficient alone. Furthermore, Ioannidis and colleagues (2014) showed that although the majority of publishing scientists have some years in which they do not publish, those that publish every year have higher citations, suggesting that high citation impact is not due to publishing rate alone.
. | Women . | Men . | . | . | . | . |
---|---|---|---|---|---|---|
Performance Metric . | Mean . | Standard error . | Mean . | Standard error . | t(1510) . | Percentage difference (F – M) . |
Publicationsa | 32.6 | 1.15 | 46.6 | 1.18 | 7.28* | –30 |
Publications (last 5 years) | 17.2 | 0.55 | 24.0 | 0.59 | 7.43* | –28 |
Publications per year | 2.6 | 0.07 | 3.4 | 0.07 | 4.08* | –24 |
Years without publications | 2.4 | 0.09 | 2.05 | 0.07 | 2.80* | +17 |
Publications per active years | 3.09 | 0.07 | 3.84 | 0.07 | 5.68* | –20 |
Citations | 962 | 56 | 1397 | 60 | 4.51* | –31 |
Citations per paper | 25.8 | 0.90 | 25.5 | 0.60 | 0.84 | +1 |
Percentage citations, self-citationsb | 8.5 | 0.30 | 10.5 | 0.20 | 4.76* | –19 |
Percentage citations, coauthor and self-citationsc | 21.82 | 0.40 | 22.62 | 0.30 | 1.13 | –3.5 |
Percent citations, coauthor onlyd | 13.28 | 0.32 | 12.14 | 0.21 | 2.81* | +8.6 |
h-index | 13.63 | 0.34 | 16.48 | 0.30 | 5.66* | –17 |
h-index (excluding self-citations)e | 12.93 | 0.32 | 15.37 | 0.29 | 5.11* | –15 |
Inflation in h with self-citations | 0.70 | 0.04 | 1.11 | 0.04 | 6.24* | –37 |
m-index (h per year) | 1.10 | 0.02 | 1.23 | 0.02 | 4.14* | –11 |
MQ-index (H-no-self-citation per active year) | 1.29 | 0.02 | 1.32 | 0.02 | 1.00 | –2 |
. | Women . | Men . | . | . | . | . |
---|---|---|---|---|---|---|
Performance Metric . | Mean . | Standard error . | Mean . | Standard error . | t(1510) . | Percentage difference (F – M) . |
Publicationsa | 32.6 | 1.15 | 46.6 | 1.18 | 7.28* | –30 |
Publications (last 5 years) | 17.2 | 0.55 | 24.0 | 0.59 | 7.43* | –28 |
Publications per year | 2.6 | 0.07 | 3.4 | 0.07 | 4.08* | –24 |
Years without publications | 2.4 | 0.09 | 2.05 | 0.07 | 2.80* | +17 |
Publications per active years | 3.09 | 0.07 | 3.84 | 0.07 | 5.68* | –20 |
Citations | 962 | 56 | 1397 | 60 | 4.51* | –31 |
Citations per paper | 25.8 | 0.90 | 25.5 | 0.60 | 0.84 | +1 |
Percentage citations, self-citationsb | 8.5 | 0.30 | 10.5 | 0.20 | 4.76* | –19 |
Percentage citations, coauthor and self-citationsc | 21.82 | 0.40 | 22.62 | 0.30 | 1.13 | –3.5 |
Percent citations, coauthor onlyd | 13.28 | 0.32 | 12.14 | 0.21 | 2.81* | +8.6 |
h-index | 13.63 | 0.34 | 16.48 | 0.30 | 5.66* | –17 |
h-index (excluding self-citations)e | 12.93 | 0.32 | 15.37 | 0.29 | 5.11* | –15 |
Inflation in h with self-citations | 0.70 | 0.04 | 1.11 | 0.04 | 6.24* | –37 |
m-index (h per year) | 1.10 | 0.02 | 1.23 | 0.02 | 4.14* | –11 |
MQ-index (H-no-self-citation per active year) | 1.29 | 0.02 | 1.32 | 0.02 | 1.00 | –2 |
otal publications within the 15 year period. bThe percentage of citations that were from the focal author themselves. bThe percentage of citations by the author and all their coauthors, representing citations by the publishing network. cPublications by coauthors, not including those included the focal author. dExcluding only those citations by the focal author. Significant effects (p < .003, since Bonferroni correction applied).
. | Women . | Men . | . | . | . | . |
---|---|---|---|---|---|---|
Performance Metric . | Mean . | Standard error . | Mean . | Standard error . | t(1510) . | Percentage difference (F – M) . |
Publicationsa | 32.6 | 1.15 | 46.6 | 1.18 | 7.28* | –30 |
Publications (last 5 years) | 17.2 | 0.55 | 24.0 | 0.59 | 7.43* | –28 |
Publications per year | 2.6 | 0.07 | 3.4 | 0.07 | 4.08* | –24 |
Years without publications | 2.4 | 0.09 | 2.05 | 0.07 | 2.80* | +17 |
Publications per active years | 3.09 | 0.07 | 3.84 | 0.07 | 5.68* | –20 |
Citations | 962 | 56 | 1397 | 60 | 4.51* | –31 |
Citations per paper | 25.8 | 0.90 | 25.5 | 0.60 | 0.84 | +1 |
Percentage citations, self-citationsb | 8.5 | 0.30 | 10.5 | 0.20 | 4.76* | –19 |
Percentage citations, coauthor and self-citationsc | 21.82 | 0.40 | 22.62 | 0.30 | 1.13 | –3.5 |
Percent citations, coauthor onlyd | 13.28 | 0.32 | 12.14 | 0.21 | 2.81* | +8.6 |
h-index | 13.63 | 0.34 | 16.48 | 0.30 | 5.66* | –17 |
h-index (excluding self-citations)e | 12.93 | 0.32 | 15.37 | 0.29 | 5.11* | –15 |
Inflation in h with self-citations | 0.70 | 0.04 | 1.11 | 0.04 | 6.24* | –37 |
m-index (h per year) | 1.10 | 0.02 | 1.23 | 0.02 | 4.14* | –11 |
MQ-index (H-no-self-citation per active year) | 1.29 | 0.02 | 1.32 | 0.02 | 1.00 | –2 |
. | Women . | Men . | . | . | . | . |
---|---|---|---|---|---|---|
Performance Metric . | Mean . | Standard error . | Mean . | Standard error . | t(1510) . | Percentage difference (F – M) . |
Publicationsa | 32.6 | 1.15 | 46.6 | 1.18 | 7.28* | –30 |
Publications (last 5 years) | 17.2 | 0.55 | 24.0 | 0.59 | 7.43* | –28 |
Publications per year | 2.6 | 0.07 | 3.4 | 0.07 | 4.08* | –24 |
Years without publications | 2.4 | 0.09 | 2.05 | 0.07 | 2.80* | +17 |
Publications per active years | 3.09 | 0.07 | 3.84 | 0.07 | 5.68* | –20 |
Citations | 962 | 56 | 1397 | 60 | 4.51* | –31 |
Citations per paper | 25.8 | 0.90 | 25.5 | 0.60 | 0.84 | +1 |
Percentage citations, self-citationsb | 8.5 | 0.30 | 10.5 | 0.20 | 4.76* | –19 |
Percentage citations, coauthor and self-citationsc | 21.82 | 0.40 | 22.62 | 0.30 | 1.13 | –3.5 |
Percent citations, coauthor onlyd | 13.28 | 0.32 | 12.14 | 0.21 | 2.81* | +8.6 |
h-index | 13.63 | 0.34 | 16.48 | 0.30 | 5.66* | –17 |
h-index (excluding self-citations)e | 12.93 | 0.32 | 15.37 | 0.29 | 5.11* | –15 |
Inflation in h with self-citations | 0.70 | 0.04 | 1.11 | 0.04 | 6.24* | –37 |
m-index (h per year) | 1.10 | 0.02 | 1.23 | 0.02 | 4.14* | –11 |
MQ-index (H-no-self-citation per active year) | 1.29 | 0.02 | 1.32 | 0.02 | 1.00 | –2 |
otal publications within the 15 year period. bThe percentage of citations that were from the focal author themselves. bThe percentage of citations by the author and all their coauthors, representing citations by the publishing network. cPublications by coauthors, not including those included the focal author. dExcluding only those citations by the focal author. Significant effects (p < .003, since Bonferroni correction applied).
Citations and impact
Not surprisingly, citations increase with time since first publication, and men have more total citations than women have (table 1, figure 2d), although women have nonsignificantly higher rates of citation per paper (table 1). This results in higher h-index scores for men than those for women (table 1). Consistent with our hypothesis that publication patterns are related to self-confidence mediated by societal influences (Cameron et al. 2013), the higher rate of citations received by men was at least partly due to self-citation, because men cite their own work more than women do (table 1, figure 2e), confirming observations from other disciplines (Maliniak et al. 2013), thereby suggesting the pattern may be widespread. Although the average differences are small, the variance is large, with men having 0%–59% of all citations due to self-citation compared with 0%–36% among women—sufficient to have a significant career impact. Rates of self-citation are influenced by a variety of factors that differ between male and female authors, such as the size and nature of their collaborative network, which can vary in a gender-specific manner (Ozel et al. 2014). However, the rate of author and coauthor citation combined was not significantly different between men and women (table 1), and coauthors (not including the author's own citations) cited the work of their female coauthors at a higher rate than that of their male coauthors (table 1, figure 2f).
Regardless of the cause, rates of self-citation do make a difference to impact metrics, particularly the h-index (Hirsch 2005). The extent of h-index inflation due to self-citation by the author alone is surprising: 41% of all researchers had a lower h-index when their own self-citations were excluded, but this consisted of 63% of men but only 50% of women (χ2 = 23.76, p < .001), with self-citation resulting in up to 14 extra h-index points in men and up to 5 h-index points in women. Over a quarter of men (28%) had an h-index inflated due to self-citation by 2 or more units, whereas only half as many women (14%) showed similar h-index inflation (χ2 = 32.69, p < .0001). Despite previous studies concluding that h-index unit increases of greater than one would rarely occur based on self-citation (Engqvist and Frommen 2008), we found h-index inflation of up to an extreme of 14 units—enough to make a substantial career impact. Our findings debunk the widely held view that self-citations have little impact on the h-index (Engqvist and Frommen 2008) and may explain recent reports that papers (Sugimoto et al. 2013) with male first authors have more citations when the citation rate includes self-citation (Sugimoto et al. 2013). The sex-specific differences in self-citation are largely driven by a small percentage of extreme self-citers. Therefore, continuing to include self-citation in estimates of research impact influences a wide range of researchers. The highest 1% of self-citers (of which 93% are men) inflate their h-index by an average 6.5 units compared with the remaining 99%, who only have an average increase in h-index of less than 1 unit (0.91, with a range of 0–5). Even the top 10% (of which 81% are men) increase their h-index by an average 3.5 units. Therefore, including self-citation in metrics of research quality should be of concern to the vast majority of authors, not just women. However, even if these self-citers are removed, the gender differences in self-citation and h-index are still significant.
Broader implications and a solution?
Excluding self-citations reduces the differences between men and women (table 1) but does not eliminate the differences, because it fails to take into account the differences between men and women in research absences. However, accounting for both of these differences by excluding self-citations and non-research-active years delivers metrics that do not differ between men and women (table 1, figure 3). Such scores are akin to the m-index, which is the h-index adjusted for career age, usually calculated as h-index per years since first publication. Here, we use an h-index that excludes self-citations, divided by research active years (defined as having at least one publication), which we have coined the m(eQuality) or m(Q)-index. Therefore, our results falsify the idea that men persist in science because of their better innate scientific abilities (coined the Summers hypothesis; Barres 2006), supporting other recent studies (Spelke 2005, Halpern et al. 2007). We suggest that the m(Q)-index would benefit all researchers because it provides a fairer evaluation of impact. For example, there is significant geographic variation in the rate of self-citation and the number of career gaps, reinforcing that current metrics have biases resulting from assuming a traditional linear career path and related to researcher personality, not just performance (figure 4). Therefore, the inclusion of self-citations and research-inactive years contributes to broader inequity, compounding geographic disparities. Furthermore, there is geographic variation in the gender disparity in research metrics (figure 4), underscoring that variations in self-citing and career interruptions are culturally influenced.
In order to test whether the patterns extend beyond the discipline of ecology, a multidisciplinary analysis is necessary. Although publishing patterns vary markedly between disciplines, making comparisons meaningless, rates of self-citation and career gaps are relative measures and can therefore be compared. We therefore used 140 male and 140 female authors publishing in Nature and Science in 2011 and found that, as in ecology, men self-cite more than women (men, 4.8%; women, 3.3%; t(198) = 3.04, p = .003) and women have more complete years without publications (men, 2.05; women, 2.97; t(198) = 2.99, p = .003). Therefore, the pattern is not confined to ecology and should be of concern across disciplines, although more detailed analysis is required. Previous studies have similarly revealed that men consistently self-cite more than women do in other disciplines (e.g., Hutson 2006, Maliniak et al. 2013).
Leveling the playing field or creating new biases?
It could be argued that changing research metrics to reflect gender differences is an artificial construct that would differentially favor women. The most radical suggestion is to exclude self-citations. Hirsch himself recognized that self-citations should be excluded, noting that “. . . ideally one would like to eliminate the self-citations” (Hirsch 2005), and subsequent studies have shown that self-citations impact the h-index (Kelly and Jennions 2006). Self-citation is an important part of publishing, and we are not suggesting that self-citation should not occur—or even that sexually dimorphic patterns of self-citation are necessarily driven by strategic self-citation or self-promotion by men. Although strategies of self-promotion occur, they can be difficult to distinguish from legitimate self-citation (Petersen et al. 2014). However, at least part of the pattern appears to be driven by a reticence by women to cite their own work. We also do not advocate that authors should not self-cite and instead suggest that women should be encouraged to self-cite more frequently where appropriate, because self-citation may have more subtle effects on citation rates, such as enhancing reputation or a form of advertisement (Petersen et al. 2014), which are more difficult to measure. However, the author's own self-citations do not indicate scientific impact. Therefore, removing author self-citations would provide a more accurate as well as more equitable estimate of publication impact.
Removing research inactive periods from career assessment is also sensible and often attempted in job recruitment and promotion as well as grant proposal evaluation. Our research reinforces the importance of ensuring that this is a vital part of research evaluation metrics. Removing research-inactive years has the greatest impact on researchers who are productive in their research-active years (figure 5). Unproductive researchers might be unfairly advantaged by removing years without publications, but among our authors, those who were unproductive tended to be less productive in all years, including those when they produced a publication and therefore stayed ranked lowly even when we removed those years in which they had no publications. The researchers who particularly benefitted were those who were productive in most years but had years in which they produced nothing. Using more accurate assessments of research-inactive periods would further enhance equitable comparisons. Like any metric, there is the possibility of gaming the metric. For example, authors might attempt to concentrate periods of publication into particular years, leaving entire years free from publications. Although possible, the gaming of publication time by an author is more difficult than other factors because of differences in submission-to-publication times between journals, with many factors outside the control of the author.
Removing self-citations and assessing output based on research-active time would provide a more level playing field to all researchers, particularly women, men who take periods of paternal leave, and those that have followed a nontraditional path into academia. This may be particularly important to researchers in their early careers, when many women leave academia and when employers are using metrics to predict future scientific success (Acuna et al. 2012), and small advantages can dramatically influence career trajectory (Merton 1968). Although issues that need to be addressed remain, we advocate that the most important first step is to ensure equality in presumed objective measures of the scientific achievement of researchers. Continued use of the h-index should at least exclude the author's own self-citations to be equitable. The current use of self-citations in impact metrics undermines academia as a meritocracy.
We thank the many people who we have discussed these ideas with over the years and who contributed to the development of ideas. In particular, Chris Johnson improved the manuscript with his suggestions.
Supplemental material
The supplemental material is available online at Supplementary Data.
References cited