National Center for Science and Engineering Statistics

Purpose. The NPRA module of the ABS collects information on R&D performed or funded by 501(c) nonprofit organizations in the United States. The nonprofit sector is one of four sectors (business, government, higher education, and other private nonprofit) that fund or perform R&D.

NCSES combines nonprofit sector data with data from the other sectors to estimate total national R&D expenditures. Results of the research activities data collected from nonprofit organizations will be used to report updated, valid, and reliable estimates of U.S. nonprofit R&D in National Patterns of R&D Resources and the Bureau of Economic Analysis system of national accounts.

The data collected will also be incorporated into the National Science Board’s biennial report, Science and Engineering Indicators. The R&D data from the nonprofit module will be reported in the Organization for Economic Cooperation and Development (OECD) periodic publications and used for international comparisons of R&D efforts. NCSES also anticipates professional associations will use data from the nonprofit R&D module. Likely users in this category include, but are not limited to, the Science Philanthropy Alliance, the Association of Independent Research Institutes, and the Health Research Alliance.

Data collection authority. Title 13, United States Code, Sections 8(b), 131, and 182; Title 42, United States Code, Section 1861-76 (NSF Act of 1950, as amended); and Section 505 within the America COMPETES Reauthorization Act of 2010, authorize this collection. Sections 224 and 225 of Title 13 require mandatory response. Office of Management and Budget No. 0607-1004.

Survey sponsor. NCSES within NSF.

Survey collection and tabulation agent. The survey is conducted annually by the Census Bureau in accordance with an interagency agreement with NCSES.

Frequency. Annual.

Initial survey year. 2021. (The questions in the FY 2020 module were first developed and used on the FY 2016 Nonprofit Research Activities Survey, which was the first national survey of R&D activities in the U.S. nonprofit population since 1997. Due to the differences between the previous surveys and the new module, 2021 is considered the initial survey year for the annual series.)

Reference period. Fiscal year 2020. The fiscal year referred to throughout this report was the nonprofit organizations’ fiscal year; for the majority (62%) of organizations reporting R&D performance, this fiscal year ended in either September or December of 2020.

Response unit. Organizations.

Sample or census. Sample.

Population size. 48,500 organizations.

Sample size. 8,000 organizations.

Target population. Included are all businesses with at least one in-scope location filing Internal Revenue Service (IRS) Form 990 as a tax-exempt organization and with FY 2019 annual payroll of $500,000 or more.

Sampling frame. The sampling frame was constructed from the final 2019 Business Register and the Exempt Organizations Business Master File Extract (EO BMF). The Business Register is the Census Bureau’s comprehensive database of U.S. businesses. Business Register data are compiled from a combination of business tax returns, data collected from the economic census, and data from other Census Bureau surveys. The Business Register includes sole proprietorships, partnerships, and corporations reporting business activity to the IRS. The EO BMF is a publicly available list from the IRS of all organizations that are exempt from filing federal income taxes.

The Business Register contains establishments that are out of scope for the nonprofit R&D module. These establishments are removed from the sampling universe. They include:

• Establishments engaged in the following North American Industry Classification System (NAICS) industries:
• 110000, 111, or 112 - Agriculture production
• 482 - Railroads
• 491 - U.S. Postal Service
• 521 - Monetary authorities–central bank
• 525 - Funds, trusts, and other financial vehicles
• 8131, 8134 - Religious and civic organizations
• 814 - Private households
• 92 - Public administration
• Establishments located in American Samoa, Northern Mariana Islands, Guam, Puerto Rico, or the Virgin Islands
• Establishments belonging to foreign entities
• Government establishments
• Establishments with zero payroll and zero employment
• For-profit establishments or nonprofit establishments within multi-establishment firms that have a majority of firm payroll in for-profit establishments

Information on industry classification, receipts, payroll, and employment was extracted from the Business Register during the frame construction. Nonprofit status for each establishment was determined by matching the Business Register to the IRS nonprofit list.

The ABS sample is selected at the firm level, so firms with multiple establishments were combined into one firm-level unit for sampling. Firm payroll, receipts, and employment are set to the respective sums across all establishments within the firm. Firm industry is set to the 6-digit NAICS code with the highest aggregate payroll within the highest 4-digit NAICS within the highest 3-digit NAICS within the highest 2-digit NAICS sector. Firm industry is set to sector 55 (management of companies and enterprises) only if there are no establishments within the firm that belong to other sectors. Firm National Taxonomy of Exempt Entities (NTEE) code is set to the NTEE code with the highest aggregate payroll within the NTEE group (first digit of NTEE code) with the highest aggregate payroll.

After firm-level units are created and firm-level codes are assigned, firms are removed from the sampling frame if any of the following are true:

• Primary activity is education services (NAICS sector 61 or NTEE code starting with B).
• The firm is also in the Business Enterprise Research and Development Survey (BERD) sampling frame.
• Firm payroll is less than $500,000.

All records in the nonprofit sample universe are assigned a primary nonprofit activity stratum as follows:

• If the organization’s NAICS code begins with “622” or the first 2 characters of its NTEE is “E2” then its stratum is set to “A” (Hospitals).
• Else if the NAICS code begins with “5417” then its stratum is set to “C” (Science and technology).
• Else if the NAICS code begins with “62” or the first character of NTEE equals “E” then its stratum is set to “B” (Other healthcare).
• Remaining unclassified organizations have their stratum set to “D” (Other).

Sample design.

The nonprofit R&D frame is stratified by state and primary type of organization (hospitals, other health care, science/technology, and all others). Within these strata some nonprofits were selected with certainty based on the following criteria:

• firms with known R&D activity (1,000 organizations)
• firms larger than stratum-specific payroll cutoffs (1,200 organizations)

The nonprofit R&D sample consisted of 8,000 organizations; 2,200 were selected with certainty.

The remaining 5,800 noncertainty cases were selected using the systematic stratified random sample selection. The maximum sample weight was 8.3.

Data collection. The survey was mailed to 8,000 nonprofit organizations in July 2021. Organizations were sent a letter informing them of their requirement to report under Title 13, United States Code, Sections 224 and 225. The letter also provided instructions on how to access the survey and submit online. There were three mail follow-ups conducted to increase response. The third mail follow-up included a paper questionnaire for select nonrespondents.

Additionally, the Census Bureau conducted email follow-ups to respondents who logged into the electronic system but did not submit the questionnaire. The collection period closed in January 2022.

Mode. The data were collected using both a paper form and the electronic instrument.

Response rates.

Check-in rate. The check-in rate is defined as the unweighted number of surveys that were submitted online by in-scope organizations, divided by the unweighted total number of all in-scope organizations in the sample. Response to individual questions did not factor into this metric. At the close of the collection period in January 2022, there were 6,800 responses submitted. Of those, 99% reported online and 1% reported using the paper form.

Unit response rate (URR). Unit response is defined as an organization providing total expenses or employment and answering the R&D performance and R&D funded questions. URR is the ratio between the number of unit respondents in a sample (numerator) and total sample size (denominator), expressed as a percentage.

For the nonprofit R&D module, the URR was 83%.

Item response rates. The distribution of values reported by sample organizations in the nonprofit module is highly skewed. Thus, rather than report unweighted item response rates, total quantity response rates are calculated, which are based on weighted data.

Total quantity response rate (TQRR). For a given published estimate other than count or ratio estimates, TQRR is the percentage of the weighted estimate based on data that were reported by units in the sample or on data that were obtained from other sources and were determined to be equivalent in quality to reported data and weighted only by sampling but not nonresponse weights. The TQRR for total expenditures for R&D performed by nonprofit organizations in the United States in 2020 was 77%.

Total quantity nonresponse rate (TQNR). For a given published estimate, TQNR, defined as 100% minus TQRR, is calculated for each tabulation cell from the nonprofits, except for cells that contain count or ratio estimates. TQNR measures the combined effect of the procedures used to handle unit and item nonresponse on the weighted nonprofits estimates. Detailed imputation rates are available upon request.

Data editing. Prior to tabulating the data, response data were reviewed and edited with both automated and manual procedures to correct reporting errors. R&D data were tabulated for records reporting $50,000 or more in R&D expenditures.

Survey analysts reviewed the R&D reported by the respondents. Research was done by evaluating the reported R&D to expenses ratio and the organization website information.

Additional data errors were detected and corrected using an automated data edit system designed to review the data for reasonableness and consistency. The editing process interactively performed corrections by using standard procedures to fix detectable errors. Quality control techniques were used to verify that operating procedures were carried out as specified.

Imputation.

Item nonresponse. If detailed R&D data was not reported by a nonprofit and could not be inferred by survey analysts, it was imputed in the same ratio as reported by other nonprofits in the same sample stratum. These imputations are reflected in the reported imputation rates.

Unit nonresponse. Estimates produced from the ABS include adjustments to account for organizations that did not respond to the survey (unit nonresponse). If available, data from public tax filings, annual reports, or audits were used to impute expenses and R&D for select nonprofits known to have performed large amounts of R&D based on public information or prior surveys. Otherwise, unit nonresponse is handled by adjusting weighted reported data as follows. Each organization’s sampling weight is multiplied by a nonresponse adjustment factor. To calculate the adjustment factors, each organization in the sample that is eligible for tabulation is assigned to the adjustment cells. The adjustment cells for nonprofits are based on certainty or noncertainty sampling strata and NAICS sector. For NAICS sector, there are three categories: healthcare, science and technology, and all other. For a given adjustment cell, the nonresponse adjustment factor is the ratio of the sum of the sampling weights for all organizations in the cell to the sum of the sampling weights for all organizations in the cell with reported data. For the nonresponse adjustment, an organization is considered a respondent if it satisfies the definition of response as available in the URR section above.

Weighting. The survey data are weighted for sampling and unit nonresponse.

Industry classification. Nonprofits are classified into one of four types of organization at the time of sampling: hospitals, other healthcare, science and technology, and all other organizations. Organizations tabulated based on the classification at the time of sampling with hospitals and other healthcare were collapsed into one tabulation group called healthcare. Classification is based on both the 2017 NAICS (https://www.census.gov/naics/) and the NTEE. NTEE code is not available for all organizations.

Organizations with more than one domestic establishment are assigned a single industry classification using a hierarchal system based on the largest payroll. For NAICS, the hierarchy is largest payroll sector, largest payroll 3-digit NAICS (within the largest sector), largest payroll 4-digit NAICS (within the largest 3-digit), and largest payroll 6-digit NAICS (within the largest 4-digit). For NTEE the hierarchy is largest first letter of the NTEE code then largest full 3 characters of the NTEE code.

Organizations are first classified as hospitals if the first two characters of their NTEE code are “E2” or the first three characters of their NAICS code are “622.” Remaining organizations are classified as science and technology if the first four characters of their NAICS code are “5417.” Remaining organizations are classified as other healthcare if the first character of their NTEE code is “E” or the first two digits of their NAICS code are “62.” All remaining organizations are classified as other.

Variance estimation.

This survey uses delete-a-group jackknife variance estimator. Note that certainty cases do not contribute to the sampling variance. The delete-a-group jackknife variance estimator requires that every sampling stratum contains at least two sampled firms. Sampling strata that do not meet this requirement are collapsed as needed to create a new set of variance estimation strata that satisfies this requirement.

Detailed relative standard errors are available on request.

The estimates produced from the NPRA module are subject to both sampling and nonsampling errors.

Sampling error. The sampling error is described above in the variance estimation section.

Coverage error. Coverage error occurs when the frame fails to completely enumerate the population of interest. There can be both undercoverage error, where units are not included in the frame, and overcoverage error, where units included in the frame are out of scope for the population of interest. The ABS uses the prior-year Business Register to construct the frame so any changes in businesses that would change the inclusion or exclusion of the business to the survey scope could be sources of coverage error. Prior to tabulation, the survey unit’s information is updated with the most recent available Business Register data to mitigate this source of error.

Nonresponse error. Nonresponse error refers to the differences in key estimates between units (i.e., organizations) in the sampling frame that were sampled for data collection and those that responded. For unit nonresponse, multiple follow-ups were conducted with nonresponding organizations, and multiple contact and data collection modes were used (i.e., phone, mail, and e-mail) to mitigate nonresponse error. The final survey weights incorporated nonresponse adjustments to reduce the risk of nonresponse bias in the final estimates. 

Nonresponse bias for survey estimates cannot be directly measured. However, the impact of nonresponse can be incorporated into the variability of survey estimates assuming the data is missing at random. Missing at random for this survey means that, conditional on the nonresponse adjustments, the propensity for an organization to respond is not related to R&D performance.

For item nonresponse, organizations were encouraged to report estimates of expenditures when actual dollar amounts could not be provided. This approach reduces item nonresponse error risk but may introduce measurement error. Imputation was conducted to help mitigate item nonresponse error.

Measurement error. The most common source of measurement error was reporting in different units (e.g., reporting in whole dollars rather than in thousands of dollars). This was corrected during data processing. Another source of error involved incorrect inclusion of organizations already represented in different R&D data collections. The R&D of these respondents was set to 0 if it was determined their R&D was already represented in other R&D survey responses. These cases included nonprofit organizations managing federal laboratories and some university-affiliated hospitals.

The questions in the FY 2020 module were first developed and used on the FY 2016 Nonprofit Research Activities Survey—which was the first national survey of R&D activities in the U.S. nonprofit population since 1997. Due to the differences between the previous surveys and the new module, data are not comparable for trend analysis.

• Employment. Paid employment consists of full- and part-time employees, including salaried officers and executives of corporations who were on the payroll in the pay period including 12 March 2021. Included are employees on sick leave, holidays, and vacations; not included are proprietors and partners of unincorporated businesses.
• Research and experimental development (R&D). R&D is planned, creative work aimed at discovering new knowledge or devising new applications of available knowledge. This includes (1) activities aimed at acquiring new knowledge or understanding without specific immediate commercial applications or uses (basic research); (2) activities aimed at solving a specific problem or meeting a specific commercial objective (applied research); and (3) systematic use of research and practical experience and resulting in additional knowledge, which is directed to producing new or improved goods, services, or processes (experimental development). R&D includes both direct costs, such as salaries of researchers and administrative and overhead costs clearly associated with the organization’s R&D. However, R&D does not include expenditures for routine product testing, quality control, and technical services unless they are an integral part of an R&D project. R&D also does not include market research; efficiency surveys or management studies; literary, artistic, or historical projects, such as films, music, or books and other publications; and prospecting or exploration for natural resources.
• Basic research. Activities aimed at acquiring new knowledge or understanding without specific immediate commercial applications or uses.
• Applied research. Activities aimed at solving a specific problem or meeting a specific commercial objective.
• Experimental development. Systematic work, drawing on research and practical experience and resulting in additional knowledge, which is directed to producing new products or processes or to improving existing products or processes.