The National Science Board stated in its Vision 2030 report that “women and underrepresented minorities remain inadequately represented in S&E relative to their proportions in the U.S. population” (NSB 2020). These disparities have also been found in the publication of peer-reviewed articles (Hopkins et al. 2013). The National Center for Science and Engineering Statistics (NCSES) has undertaken research to examine linkages between publication output and careers in research (Chang, White, and Sugimoto forthcoming).

Matching publication output data to demographic survey data provides a key to understanding publication output in conjunction with authors’ demographic, training, and career information. Prior researchers have attempted to add author demographics using various methods, such as sex and race disambiguation algorithms (e.g., NamSor, Ginni, Ethnicolr, OriginsInfo), that estimate the probability of race or sex from given names (or, in the case of Face⁺⁺, from images). The accuracy of these matches varies dramatically by country and field; sex disambiguation algorithms perform better for western countries and poorly for other countries, specifically in Asia and South America (Karimi et al. 2016). In addition, some scientific fields, such as astronomy and astrophysics, generally use initials rather than given names. Despite these limitations, researchers have observed sex and race disparities in publication output (Hopkins et al. 2013; Larivière et al. 2013; Marschke et al. 2018; and NSB Indicators 2018: S&E Publication Patterns, by Gender).

The limitations associated with the earlier approaches can be overcome using data directly collected from the authors. One such source is the NCSES Survey of Doctorate Recipients (SDR),^* which provides demographic, education, and career history information from a sample of individuals with a U.S. research doctoral degree in a science, engineering, or health field (NCSES 2021). Clarivate, the architect of Web of Science (WoS),^† matched SDR respondents to publication records in the WoS publication output database. The results provide demographic information, such as sex and race or ethnicity of publication authors.

These data shed light on publication output differences between groups defined by race or ethnicity and sex, by discipline, and by impacts to R&D career paths (Chang, White, and Sugimoto forthcoming).^‡ The point estimates in Figure PBS-A show the odds of pre-doctorate student publishing by ethnic group or sex relative to White students (or men, for the sex comparison) while the error bars show the confidence around that point estimate (95% confidence interval). The confidence interval is closely linked to the size of the sample. In the SDR-WoS data, the number of minorities and women receiving degrees in the population influences the sample size—and, consequently, the ability to measure odds ratios. For example, there are 3,750 women who received mathematics or statistics PhDs compared to 10,450 men (Table SPBS-32). A similar issue arises for mathematics or statistics PhDs by race or ethnicity (Table SPBS-33). Overall, compared to White graduates, Asian, Black, or Hispanic graduates are less likely to publish before their doctorate in biological, agricultural, and other life sciences; engineering; health sciences; and social sciences.

Compared to men, women are less likely to publish before graduation in the biological sciences, agriculture, engineering, health sciences, physics, and social sciences. Pre-doctorate publications appear to factor into obtaining a job in which research is the primary activity.^§ For those with at least one pre-PhD publication, 56% reported that their first job has research as its primary activity compared to 37% of those without a publication (Chang, White, and Sugimoto forthcoming).

* A machine learning approach matches the SDR respondents to the authors of publications indexed by the Web of Science (WoS). The matching algorithm incorporates name commonality, research field, education, employment affiliations, coauthorship network, and self-citations to predict matches from the SDR respondents to the WoS.

^† WoS is a bibliometric database of conference proceedings and peer-reviewed literature with English-language titles and abstracts.

^‡ To predict pre-doctorate publishing propensity, separate models were fitted for each doctoral field, and the following factors from the NCSES’s Survey of Earned Doctorates were controlled: doctorate award year, type of PhD-awarding institution, source of primary support, community college experience, U.S. citizenship status at the time of degree award, level of parental education, marital status, dependents under 18 years old, disability, graduate debt, and name commonality.

^§ The model controls for critical factors, such as the PhD institutions ranking as a high research institution, year of graduation, citizenship, parental degree, and student debt. The model does not measure article submissions or rejections.

This sidebar uses cross-disciplinarity as an envelope term that includes convergent, multidisciplinary, and interdisciplinary research because the measurement techniques for examining publication output are similar. Cross-disciplinary research includes the following:

• Convergent research that is driven by a specific and compelling problem requiring deep integration across disciplines (NSF 2019). Convergent science is a team-based approach to problem solving cutting across fields of inquiry and institutional frontiers to integrate areas of knowledge from multiple fields to address specific scientific and societal challenges.
• Multidisciplinary research (MDR) that “juxtaposes two or more disciplines focused on a question … [where] the existing structure of knowledge is not questioned” (NRC 2014:44).
• Interdisciplinary research (IDR) that “integrates information, data, methods, tools, concepts, and/or theories from two or more disciplines focused on a complex question, problem, topic, or theme” (NRC 2014:44).

Efforts using publication output to measure cross-disciplinary research yields results that are not suitable for comparing at the country level (Wagner et al. 2011; Wang and Schneider 2020). This finding is similar to sidebars in previous Indicators reports (NSB 2010; NSB 2016; Wagner et al. 2009). This sidebar explains the ongoing methodological issues with measuring convergence, MDR, and IDR at the country level and provides potential directions for future research.

For measurement at the country level, researchers have analyzed cross-disciplinary research using various bibliometric measures. Some have used article citations (Campbell et al. 2015; Porter and Chubin 1985), coauthor fields of specialization (Porter et al. 2007), text mining of abstracts or keywords listed on each article (Del Rio et al. 2001), or network analysis (Leydesdorff and Rafols 2011). An analysis of various approaches for measuring interdisciplinarity revealed a lack of consistent measurement outcomes across scientific fields, over time, and for countries or economies (Digital Science 2016).

Measuring cross-disciplinarity is challenging because indicators that are valid by one measure (e.g., citation counts), are not stable in another scientific area. For example, looking within the broad field of health sciences, health economics uses fewer citations, while biomedicine uses many more. When attempting to measure cross-disciplinarity for health sciences, the differences between health economics and biomedicine are, at least in part, related to different citation habits and not necessarily to differences in the cross-disciplinarity of the research.

Although research has not uncovered robust cross-disciplinary measures for countries, there are insights into the growth and influence of convergence, IDR, and MDR. Measured broadly, researchers find growth in cross-disciplinarity: “from about the mid-1980s, both natural sciences and engineering (NSE) and medical fields (MED) raised their level of interdisciplinarity at the expense of a focus on specialties” (Larivière and Gingras 2014:197). The team also found that the social sciences, as well as the arts and humanities, were the most open to collaborating with other disciplines. While cross-disciplinarity has grown, citation lags are associated with cross-disciplinary research papers. Specifically, they garner fewer than the normal number of citations for the first 3 years but pick up more citations than normal over 13 years (Wang, Thijs, and Glänzel 2015).

Recently, Digital Science prepared a report for the Research Councils of the United Kingdom (RCUK) that scanned the current literature and measurement approaches (Digital Science 2016). RCUK concluded that “no single indicator of interdisciplinarity (either MDR or IDR) analysed here should, used alone, satisfy any stakeholder. They show diverse inconsistency—in terms of change over time, difference between disciplines and trajectory for countries—that raises doubts as to their specific relevance” (Digital Science 2016:8). The RCUK report suggested that combining bibliometric IDR measures with other data, such as award information, could create a framework for expert analysis of IDR. Among the recommendations were continued exploration of text analysis and the inclusion of departmental affiliations in award information.

Similarly, the 2021 National Academies of Sciences, Engineering, and Medicine workshop on Measuring Convergence in Science and Engineering found that “using a single or even a few atomistic indicators to measure complex research activities capable of addressing societal problems is misguided” (NASEM 2021:49). Workshop participant Ismael Rafols suggested shifting from an atomistic to a portfolio approach, investigating the entire landscape that makes convergence possible.