Assessment of the FY 2016 Survey of Nonprofit Research Activities to Determine Whether Data Meet Current Statistical Standards for Publication

NCSES has a set of statistical standards for the release of “official statistics,” specifically

Standard 9.2: The statistical quality of official statistics must undergo rigorous program review and statistical review and approved by the chief statistician for releasing.

Guideline 9.2a: The reliability of official statistics must meet the following quality criteria:

• Top line estimates have a coefficient of variation (CV) < 5%.
• The estimated CV for the majority of the key estimates is ≤ 30%.

Guideline 9.2b: The indicators of accuracy of official statistics must meet the following quality criteria:

• Unit response rates >60%.
• Item response rates >70%.
• Coverage ratios for population groups associated with key estimates are >70%.
• Above thresholds may not apply if nonresponse bias analyses are at an acceptable level.

The NCSES standards focus on the aspects of accuracy, including response rates, data missingness, and frame coverage as well as precision. These elements align with the accuracy and reliability dimension of the objectivity domain. Demonstrating that the NPRA Survey produces accurate and reliable estimates of total R&D performing and funding is the primary focus of this assessment. Therefore, we summarize the metrics as compared to the standard in table 1 and follow with a more detailed description for each standard.

CV. The total number of responding organizations is 3,254. When evaluating total expenses, the CV is 0.5%, well below the 5% CV standard. However, the sample was designed to optimize total expenses, so other estimates are expected to have higher variability. The CVs for the proportion of performing organizations and proportion of funding organizations are 12%. The high CVs are largely due to the low proportions of nonprofits that reported performing and funding R&D, 6% (+/−1.4%) and 4% (+/−1.0%), respectively. However, the confidence intervals are small for both of these estimates. The CVs for the mean performance and mean funding are both 8%. These CVs, based on a subset of organizations that indicated they perform and fund research, meet the CV standard of 30% for the majority of key estimates but not the CV standard of 5% for the top line estimates. Since imputation is a separate criterion, these CV estimates are based only on sampling variance. Table 2 includes additional CVs for total performance and funding amounts by source, R&D type, and field as well as the mean number of full-time equivalents. All but nine of the estimates meet the standard of a 30% CV for key estimates.

Coverage ratio. Based on analysis conducted during the sampling, 85% of performers and 88% of funders are estimated to be covered in the frame, which is well above the NCSES threshold of 70% coverage. The frame coverage estimates are based on an analysis of the percentage of “likely” performers and “likely” funders on the frame after exclusions based on type of organization (e.g., educational institutions or churches) and the size truncation based on total expenses for public charities or total assets for private foundations. The exclusions and truncation were conducted to increase the efficiency of finding R&D performers and funders. Since the exclusions focused on organization types that are not likely to perform or fund or are small in terms of overall expenses or assets, the undercoverage most likely has little impact on the estimated total R&D performance and funding.

Unit response rate. The unweighted response rate (48%) falls short of the 60% standard. The weighted response rate (61%), which uses the sampling weights for all 6,373 sample organizations (6,071 identified as eligible), does meet the standard. The difference in the unweighted and weighted response rates means that the average weight for responders (mean weight = 23.6) is higher than that for nonresponders (14.2). Unweighted and weighted response rates provide distinct measures of survey quality. Unweighted response rates are an indicator of success with data collection operations.^{​For more information on lessons learned regarding data collection operations, see the Working Paper, FY 2016 Nonprofit Research Activities Survey: Summary of Methodology, Assessment of Quality, and Synopsis of Results at https://www.nsf.gov/statistics/2021/ncses21202/#chp5.} Weighted response rates are most useful when the sampling fractions vary across strata in a sample design and when the interest is population-level inference about the amount of information captured from the population sampled.

Both unweighted and weighted response rates are important metrics for evaluating the quality of the data. Although the weighted response rate meets the standard, the unweighted response rate does not. Therefore, additional analysis measuring the impact of nonresponse is required.

Item response rate. The amount of imputation for performing status (28%) and funding status (17%) meets the minimum threshold for item response, as does the imputation for funding amount (26%). However, the performing amount is just below the standard with 32% of the responses imputed (289 of 912).

Nonresponse bias. The response rate and the item response rate evaluations both include metrics that do not meet the NCSES quality standards. The standards state that the nonresponse standard threshold may be exceeded if the estimate for nonresponse bias is acceptable. The nonresponse bias is a function of the weighted response rate and the weighted mean difference between the respondents and nonrespondents is as follows:

where

is the weighted response rate,

is the weighted respondent mean, and

is the weighted nonrespondent mean.

The estimated nonresponse bias was a mean bias of −4 million for total expenses (overall mean = 13, respondent mean = 9). This was a relative bias of −0.451 (table 3). This result suggests that organizations with higher amounts of expenses were less likely to respond than those with lower total expenses. This is confirmed from the response rates for deciles based on total expenses presented in table 4. The lowest response rates occur in deciles 9 and 10, the organizations with the highest amount of expenses. Over 50% of the organizations in these deciles are hospitals, the organization type experiencing lowest response. When removing organizations in deciles 9 and 10, the relative bias is decreased to −0.112 (data not shown), demonstrating that the largest organizations contribute the majority of bias in the total.

The imputation and nonresponse adjustments were informed by the nonresponse bias analysis. The imputations were prioritized based on previous data from the pilot survey as well as administrative data for the largest organizations. These substitutions reduced the bias of total expenses to −0.336. Further, the imputation model predicting the amount of performance and funding based on total expenses reduces the nonresponse bias to −0.169 for total expenses.

To address higher nonresponse for hospitals and larger organizations, the nonresponse adjustment cells included hospitals and a flag identifying the largest 500 organizations in terms of expenses. The nonresponse adjustment was based on a ratio adjustment using total expenses. Therefore, the bias is 0 when evaluating weighted nonresponse bias in terms of total expenses. Further, if successful, the nonresponse adjustment based on expenses will reduce the nonresponse bias for correlated measures such as assets and revenues. The relative bias in assets (−0.403) and revenues (−0.414) are similarly high based on the full and partially completed surveys. After weighting, the nonresponse bias is reduced to +0.046 for total assets and +0.014 for total revenues.

Considering that hospitals and health organizations tended to have lower response rates, an indicator for hospitals and non-hospitals was included when defining the nonresponse adjustment cells. The result was low relative bias for health and medical organizations when measuring total expenses (−0.04), total assets (−0.03), and total revenues (−0.04) (data not shown).

In summary, although the nonresponse bias is high, the information available on the nonrespondents allows for imputation and nonresponse adjustments to address these observed biases. Assuming R&D performance and funding expenditures are correlated with the size of the organization based on expenses, the risk of nonresponse bias is reduced. This is the case when measuring assets and revenues, which were not used in the weighting but are both correlated with expenses. We cannot directly measure the reduction in the bias for total performing and total funding since we do not have data for nonrespondents. However, we can evaluate the correlation of expenses and R&D performance and funding using the respondents to the survey. Table 4 provides the mean R&D performance and R&D funding for each expense decile. Both increase as the deciles of total expenses increase. The log-log imputation models used for the NPRA Survey are further evidence of a relationship between expenses and R&D performance and funding, where total expenses was a significant predictor for both models, 0.54 (standard error = 0.08) for total performing and 0.47 (standard error = 0.09) for total funding.