Statistical Definition of Development Clarified: Effect on Reported Federal R&D Totals
In recent years, several efforts have taken place to better harmonize the statistical definition of federal spending on research and development. There are two main sources of official information on federal government R&D funding, namely, the “Research and Development” chapter in the Analytical Perspectives report accompanying the president’s annual proposal for the Budget of the United States Government (often identified as “the President’s Budget”), issued by the Office of Management and Budget (OMB), and the Survey of Federal Funds for Research and Development from the National Center for Science and Engineering Statistics (NCSES). These data sources are definitionally consistent with each other, and each is definitionally consistent with international guidelines for reporting R&D funding and performance, per the Organisation for Economic Co-operation and Development (OECD) Frascati Manual. While the information in the President’s Budget is presented as fiscal year budget authority, the NCSES statistics are presented as fiscal year obligations. However, in both data sets, a substantial decline in R&D is evident, beginning with FY 2017 budget authority per OMB and with FY 2016 obligations statistics per NCSES. Overall, this decline is not the result of a change in R&D policy per se but, rather, a refinement in the definition of R&D. This InfoBrief discusses this definitional change and provides context on the effect of this change on federal R&D statistics.
Refinement in the Definition of Development
Each year OMB issues an update to Circular A-11 Preparation, Submission, and Execution of the Budget, providing detailed guidance to federal agencies on the budget formulation process. Following the release of the seventh edition of the OECD Frascati Manual in October 2015, which clarified the definition of “development” for the purposes of statistical data collection,OMB updated the definition of R&D for federal agencies in the July 2016 release of Circular A-11 for the FY 2018 budget formulation process. As part of this update, OMB added additional guidance for agencies to differentiate and exclude “preproduction development” from experimental development, citing the pre-production category of the Department of Defense (DOD) Research Development Test and Evaluation (RDT&E)—specifically, Budget Activity 7 for operational system development (OSD):
Pre-production development, which is defined as non-experimental work on a product or system before it goes into full production, including activities such as tooling and development of production facilities. For example, exclude activities and programs that are categorized as “Operational Systems Development” in DOD’s budget activity structure. Activities and programs of this type should generally be reported as investments in other major equipment.
Pre-production development, or OSD for DOD, activities that were previously reported in R&D as development activities were now excluded under the revised guidance for experimental development. Although the Frascati Manual always referred to the “D” in R&D as experimental development, the issue of pre-production development was made more explicit with the 2015 edition of the manual, effectively moving the topic from what was an annex to the 2002 edition directly into the definition and exclusions presented in the 2015 manual.
Effect on Federal R&D Obligations Statistics
For an agency to incur an obligation for R&D, that agency should have been granted R&D budget authority. Therefore, the NCSES Survey of Federal Funds for Research and Development is tied to the universe of agencies presented in the President’s Budget. OMB’s Circular A-11 highlights this connection not just in terms of the agencies themselves but also with regard to each agency’s R&D portfolio.table 2).Thus, following the release of the July 2016 update to Circular A-11, NCSES revised the Survey of Federal Funds for Research and Development FYs 2016–17 (volume 66) reporting instructions to exclude OSD from experimental development. This resulted in a noticeable change in the obligations for R&D reported by the NCSES survey between FY 2015 and FY 2016 (
Federal obligations for R&D, by type, and for operational system development: FYs 2015–18
na = not applicable.
Because of rounding, detail may not add to total. Percentages are computed using actual dollars reported.
National Center for Science and Engineering Statistics, Survey of Federal Funds for Research and Development.
Pre-Production Development Data
Although the removal of pre-production development most clearly impacted DOD’s development data, NCSES created an additional series to present both R&D and the DOD RDT&E, which includes the DOD OSD data and provides some comparability with the previous development time series.The R&D data reported by the National Aeronautics and Space Administration (NASA) were impacted as well. NASA noted that approximately $2 billion in its development obligations was no longer included in its R&D reporting effective with FY 2016 survey results. The Department of Energy (DOE) was the only other agency that would have had some possible OSD-type pre-production development obligations. However, DOE staff noted they had never included these data in their reports to NCSES. No other agencies had any R&D activities resembling the DOD OSD category.
Defense S&T, R&D, and RDT&E
Defense-related R&D activities are typically ordered into three categories, namely, science and technology (S&T) programs, R&D, and RDT&E (figure 2). To differentiate between the part of the federal R&D budget that supports science and key enabling technologies (including technologies for military and nondefense applications) and the part that primarily supports testing and evaluation (mostly of defense-related weapons systems), DOD since 1994 has reported its development obligations into two separate development categories, namely, advanced technology development (Budget Activity 3) and major systems development (Budget Activities 4–7). Thus, DOD defined its S&T budget as one that comprises basic research (Budget Activity 1), applied research (Budget Activity 2), and advanced technology development (Budget Activity 3). DOD has suggested most other federal agencies’ entire R&D programs are equivalent in nature to DOD’s S&T program. Excluding the OSD data from the R&D calculus had no impact on S&T totals, and Budget Activity 7 is not part of the DOD S&T programs. However, the exclusion of OSD did result in changes to DOD’s major system development category—and, thus, to what is reported in the various DOD R&D and RDT&E totals.
Department of Defense research, development, test, and evaluation categorization in National Center for Science and Engineering Statistics data
DOD = Department of Defense. NCSES = National Center for Science and Engineering Statistics. RDT&E = research, development, test, and evaluation. S&T = science and technology.
National Center for Science and Engineering Statistics.
The consequence of the 2016 Circular A-11 revisions is that R&D effectively is defined as a subset of RDT&E. NCSES still collects two categories for DOD’s development; beginning with volume 66 of the Survey of Federal Funds for Research and Development, however, the explicit exclusion of pre-production development resulted in changes to major systems development, only covering Budget Activities 4–6 (advanced component development and prototypes, system development and demonstration, and RDT&E management support, respectively). DOD’s Budget Activity 7 (OSD) is now excluded from DOD’s R&D totals. Because NCSES still continues to collect the DOD Budget Activity 7 obligations separately, though, its total RDT&E obligations are available as part of the survey’s data set.table 3). Further, of the $30.8 billion in FY 2018 total OSD, $22.9 billion (75%) was obligated to businesses, $7.0 billion (23%) to DOD intramural performers, and $826.0 million (2%) to all other performers, including federally funded research and development centers, higher education institutions, nonprofit institutions, and foreign performers.Of the $24.6 billion in OSD obligations in FY 2016, 72% ($17.6 billion) were reported by the Department of the Air Force. OSD obligations increased to nearly $30.8 billion in FY 2018, and the Air Force still accounted for 68% of the total, but the Department of the Navy’s share of the OSD has increased from 5.3% in FY 2016 to 8.3% in FY 2018 (
Although OMB reported totals for R&D exclude DOD Budget Activity 6 amounts, NCSES continues to report Budget Activity 6 as part of experimental development because it includes the administrative and management support costs for DOD’s S&T, R&D, and RDT&E. The OECD standard for the collection of R&D maintains that administrative and support costs to manage R&D programs are part of the cost to perform R&D and should be included.In addition, DOD’s funding for the Small Business Innovation Research and Small Business Technology Transfer programs originate from category Budget Activity 6 and these programs are within the scope of R&D.
Department of Defense obligations for research, development, test, and evaluation, by agency: 2015–18
NA = not available.
RDT&E = research, development, test, and evaluation.
a Department of Defense (DOD) development obligations have been reported in two categories, advanced technology and major systems, since volume 44 (FYs 1994–96). As of volume 66 (FYs 2016–17), the definition of major systems development was changed to represent DOD Budget Activities 4–6 instead of Budget Activities 4–7.
b Funding for DOD's Budget Activity 7 (operational system development) was first reported as a separate category for volume 66.
Because of rounding, detail may not add to total.
National Center for Science and Engineering Statistics, Survey of Federal Funds for Research and Development.
Data Sources and Limitations
NCSES provides comprehensive definitions of research and development from U.S. and international sources. For additional information regarding the definition of R&D, see Definitions of Research and Development: An Annotated Compilation of Official Sources.
The Analytical Perspectives of the President’s Budget contain analyses highlighting specific subject areas or other important aspects of the budget with additional context. Data shown in the Analytical Perspectives report’s chapter on R&D are collected directly by OMB under a specific Budget Data Request for R&D during the budget formulation process. The federal budget does not include a separately identified R&D account. Furthermore, most appropriations for R&D are not directly labeled as such (except in certain program areas, such as defense, energy, health, and environment). Thus, most funds for R&D are not line items in agency budget submissions but are included instead as part of general program funding.
To provide information on federal R&D funding, OMB requires all agencies with R&D funding levels greater than $10 million annually to submit data on their R&D programs as part of their annual budget submissions. Such agencies are requested to provide data on their funding levels for basic research, applied research, experimental development, R&D facilities, and capital equipment for R&D, in accordance with OMB’s Circular A-11, Section 84: “Character Classification (Schedule C).”
1The most recent international standard definitions and guidelines for the collection of R&D statistics can be found in Organisation for Economic Co-operation and Development (OECD). 2015. Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development. The Measurement of Scientific, Technological and Innovation Activities. Paris: OECD Publishing. R&D comprises basic research, applied research, and experimental development. These definitions reflect the international consensus of statistical organizations that collect R&D data in OECD member countries. These categories and their key features have not fundamentally changed over the years, but the associated descriptive texts have on occasion been updated and refined to better clarify the intended content of each of these types of R&D. See https://www.oecd.org/publications/frascati-manual-2015-9789264239012-en.htm.
2Budget authority means the authority provided by law to incur financial obligations that will results in outlays (see OMB Circular A-11, Section 20.4). Thus, budget authority sets a limit on new obligations an agency may incur each fiscal year. See Office of Management and Budget (OMB), Executive Office of the President. 2016. Circular A-11: Preparation, Submission, and Execution of the Budget, Section 20-4, page 20-11. Washington, DC. https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/a11_current_year/a11_2016.pdf.
3Obligations represent the amount for orders placed, contracts awarded, services received, and similar transactions during a given period, regardless of when the funds were appropriated or when future payment of money is required. Obligations include funds from direct appropriations, trust funds, special accounts, fees and charges, and other federal sources for the year of the obligation. Obligations include the full cost of R&D, both specific project costs and overhead costs. Interagency transfers for R&D are reported by the transferring agency as R&D or R&D plant, not by the agency receiving the funds.
4The 2015 Frascati Manual defines experimental development as “systematic work, drawing on knowledge gained from research and practical experience and producing additional knowledge, which is directed to producing new products or processes or to improving existing products or processes.” Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development. §1.35, page 29.
5The 2016 edition of OMB Circular A-11 is used to inform agencies on the budget formulation process for the FY 2018 proposal for the President’s Budget submission to the Congress, which was issued in 2017. OMB and the federal agencies begin work on the FY 2018 budget before FY 2016 has closed and before the full appropriations processes have been completed for FY 2017. Therefore, the FY 2017 budget authority in the 2018 proposed budget are still on the previous basis, and any revisions to the FY 2017 to account for definitional changes in experimental development do not present themselves until the issuance of the FY 2019 President’s Budget proposal.
6The DOD RDT&E budget is comprised of seven Budget Activity categories. For additional information, see Department of Defense, Under Secretary of Defense (Comptroller) (DOD/OUSD[C]). 2017. Research, Development, Test, and Evaluation Appropriations. In Department of Defense Financial Management Regulation, Volume 2B: “Budget Formulation and Presentation (Chapters 4–19),” pages 5-1–5-24. DOD 7000.14-R. Washington, DC: DOD/OUSD(C). https://comptroller.defense.gov/Portals/45/documents/fmr/Volume_02b.pdf.
7OMB Circular A-11 (2016), Section 84.2(c), pages 84-2–84-3. https://obamawhitehouse.archives.gov/omb/circulars_a11_current_year_a11_toc.
8See Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development (2015), §2.35 and §2.36. Specifically, Frascati Manual 2015 notes the importance of differentiating “experimental development” from more expansive product development (which includes commercialization) and from pre-production development, a term often used in large-scale government defense or aerospace projects, which includes non-experimental work on products or systems such as final design engineering, tooling and industrial engineering, and user demonstrations—and even low-rate initial production activities (§7.47).
9OMB. 2018. Analytical Perspectives, Budget of the United States Government, Fiscal Year 2019. Chapter 18, Research and Development, pages 233–41; Table 18-2, pages 238–40. https://www.govinfo.gov/content/pkg/BUDGET-2019-PER/pdf/BUDGET-2019-PER-7-5.pdf.
11See Congressional Research Service (CRS). 2020. Department of Defense Research, Development, Test, and Evaluation (RDT&E): Appropriations Structure. CRS Report R44711. Washington, DC: CRS. https://fas.org/sgp/crs/natsec/R44711.pdf.
12National Center for Science and Engineering Statistics. 2019. Federal R&D Funding, by Budget Function: Fiscal Years 2018–20. NSF 20-305. Table 6. Alexandria, VA: National Science Foundation. https://ncses.nsf.gov/pubs/nsf20305/.
13See OMB Circular A-11, Schedule C, Section 84.3(g), page 84-7. https://obamawhitehouse.archives.gov/omb/circulars_a11_current_year_a11_toc.
14See National Center for Science and Engineering Statistics. 2018. Federal Funds for Research and Development, Fiscal Years 2016–17. Technical Notes. Alexandria, VA: National Science Foundation. https://ncsesdata.nsf.gov/fedfunds/2016/fedfunds_2016_tech_notes.pdf.
15See National Center for Science and Engineering Statistics. Federal Funds for Research and Development. Fiscal Years 2016–17, 2017–18, 2018–19. Tables 9 and 10. Alexandria, VA: National Science Foundation. https://nsf.gov/statistics/srvyfedfunds/.
17As per the Survey of Federal Funds for Research and Development, most of DOD’s OSD funds were obligated to industry and intramural performers (see the Department of Defense obligations, by agency and performer, National Center for Science and Engineering Statistics, Federal Funds for Research and Development, 2018. Detailed Statistical Table 9. https://nsf.gov/statistics/srvyfedfunds/. Further, based on NCSES and U.S. Census Bureau interviews with defense contractors, as part of the Business Research and Development and Innovation Survey there is considerable evidence that private sector contractors (those receiving the DOD funding) do not consider a substantial share of OSD-funded activities to be R&D.
18See Frascati Manual: Guidelines for Collecting and Reporting Data on Research and Experimental Development (2015), §4.29 and §4.30.
19National Center for Science and Engineering Statistics. 2018. Definitions of Research and Development: An Annotated Compilation of Official Sources. Alexandria, VA: National Science Foundation. Available at https://www.nsf.gov/statistics/randdef/.
Pece C, Jankowski J; National Center for Science and Engineering Statistics (NCSES). 2021. Statistical Definition of Development Clarified: Effect on Reported Federal R&D Totals. NSF 21-326. Alexandria, VA: National Science Foundation. Available at https://ncses.nsf.gov/pubs/nsf21326/.
National Center for Science and Engineering Statistics
Directorate for Social, Behavioral and Economic Sciences
National Science Foundation
2415 Eisenhower Avenue, Suite W14200
Alexandria, VA 22314
Tel: (703) 292-8780
FIRS: (800) 877-8339
TDD: (800) 281-8749