Microbusiness R&D and Innovation: 2016

Purpose. The Microbusiness R&D and Innovation Survey (BRDI-M) is the primary source of information on research and development expenditures, the R&D workforce, and innovative activity of microbusinesses operating in the 50 U.S. states and the District of Columbia.

Data collection authority. The information collected by BRDI-M is solicited under the authority of the National Science Foundation (NSF) Act of 1950, as amended, and the America COMPETES Reauthorization Act of 2010. Response to this annual survey is mandatory and confidential; the U.S. Census Bureau collects the data under the authority of Title 13, Section 8 of the United States Code. The Office of Management and Budget (OMB) control number is 0607-0912 and expires on 29 February 2020.

Survey sponsors. BRDI-M is co-sponsored by the National Center for Science and Engineering Statistics (NCSES) and the U.S. Census Bureau.

Survey collection and tabulation agent. The pilot expansion of the Business R&D and Innovation Survey (BRDIS) was conducted by the U.S. Census Bureau in accordance with an interagency agreement with NCSES within NSF.

Frequency. One time; questions from BRDI-M will be incorporated into the forthcoming Annual Business Survey (ABS).

Initial survey year. BRDI-M collected data for calendar year 2016 as a pilot expansion to BRDIS, which began collection in 2008 after replacing the Survey of Industrial Research and Development (SIRD) that collected data for 1953–2007.

Reference period. Calendar year 2016.

Response unit. Company.

Sample or census. Sample.

Population size. 3,460,816 companies.

Sample size. 199,991 companies.

The survey is administered both to companies known to have performed R&D and to companies with no known history of R&D activity. BRDI-M has been designed to provide detailed statistics on domestic R&D expenditures of microbusinesses located in the United States and also statistics on these companies’ R&D employees, intellectual property, and innovation activities.

The target population for BRDI-M consists of all for-profit companies that have at least 1 paid employee but fewer than 10 paid employees in the United States, that have at least one establishment that is in business during the survey year and is located in the United States, and that are classified in certain industries based on the 2012 North American Industry Classification System (NAICS).

The Business Register, a U.S. Census Bureau compilation that contains information on more than 3 million establishments with paid employees, serves as the primary input to the sample frame from which the sample is selected. For a given company with more than one establishment, the prior year’s annual payroll and employment data for its active establishments are summed to the company level. Companies are excluded from the frame if they are classified in a NAICS industry that is outside the scope of BRDI-M or if they were selected in the 2016 BRDIS sample (to reduce response burden on these companies). Additionally, companies are excluded from the frame if they have no employees or more than nine employees, based on their prior year’s aggregated employment data.

The scope of the 2016 BRDI-M is limited to companies that (1) have a majority of their payroll allocated to establishments classified as for-profit businesses; (2) are classified within a specific set of industries as defined by NAICS; (3) have at least 1 paid employee but fewer than 10 paid employees in the United States, based on employment on 12 March 2015; (4) have at least one establishment that is physically located in the United States and is in business at the end of calendar year 2016 (the time when the U.S. Census Bureau finished the 2015 Business Register processing); and (5) are not federally funded R&D centers.

Single-unit company records were extracted from the 2015 Business Register if the company had at least 1 paid employee but fewer than 10 paid employees in 2015 or, if employment information was unavailable, the company’s 2015 payroll was greater than or equal to $50,000. Companies were removed from the sample frame if: their NAICS codes were designated as Crop production (NAICS 111), Animal production (NAICS 112), Postal service (NAICS 491), Educational services (NAICS 61), Private households (NAICS 814), or Public administration (NAICS 92); they were no longer in business; they were nonprofits; they were not located in the 50 U.S. states or the District of Columbia.

Records for active establishments from multiunit companies were extracted from the 2015 Business Register if the given establishment’s 2015 payroll was greater than zero or if the establishment employed at least one person in 2015. Prior to creating records for multiunit companies from these establishments, establishments classified as Postal service (NAIC 491), Private households (NAICS 814), or Public administration (NAICS 92) were removed, as were those that were not physically located in the 50 U.S. states or the District of Columbia. Unlike single-unit companies, establishments classified as Crop production (NAICS 111), Animal production (NAICS 112), or Educational services (NAICS 61) were not removed during the construction of multiunit company records. From the resulting set of multiunit companies, companies were removed from the sample frame if they had no paid employees, more than nine paid employees, or if the payroll associated with their nonprofit establishments was greater than the payroll of their for-profit establishments.

Each company was assigned to 1 of 60 industry-based strata for sampling based on the reported business segment in which the company performed the largest amount of total domestic R&D as reported in the prior period (2011–15 BRDIS), if available. If these business segment data were not reported for a given company, assignment is based on the NAICS codes of its establishments in the U.S. Census Bureau’s Business Register using the following method, with some adjustments made to account for vertical integration of related business activities within the company. The company was first assigned to the economic sector, defined by a 2-digit NAICS code that accounted for the highest percentage of its aggregated annual payroll. Then the company was assigned to a subsector, defined by a 3-digit NAICS code that accounted for the highest percentage of its annual payroll within the economic sector. Next, the company was assigned a 4-digit NAICS code within the subsector, again based on the highest percentage of its aggregated annual payroll within the subsector. Finally, the company was assigned a 6-digit NAICS code within the 4-digit NAICS code, based on the highest percentage of its aggregated annual payroll within the 4-digit NAICS code.

Each company in an industry-based stratum for sampling was further assigned to one of two sizes of strata based on Business Register information on the number of employees in the company: (1) companies with 1–4 employees or payroll between $50,000 and $250,000 if the employment count was missing (small stratum), or (2) companies with 5–9 employees or payroll greater than $250,000 if the employment count was missing (large stratum). Survey statistics are published for microbusinesses, those with 1–4 employees. Businesses with 5–9 employees were also sampled in order to compare microbusiness survey results with estimates from BRDIS, which also surveyed businesses with 5–9 employees. For 2016, there were 2,595,220 companies in the first stratum and 865,596 companies in the second stratum, for a total of 3,460,816.

The sample for BRDI-M consisted of nearly 200,000 companies. The sample was selected separately within the two main strata: companies with 5–9 employees (large) and companies with 1–4 employees (small). Companies in the following NAICS codes were selected with certainty: 3252, 3254, 3255, 3259, 333242, 334, 335, 336414, 336415, 336419, 336992, 5112, and 5415. Additionally, the largest 100 companies by state based on payroll and companies with a NAICS code of 5417 were selected with certainty in the small stratum. The large stratum did not have any companies in the 5417 NAICS code on the frame because all such companies were selected with certainty for the 2016 BRDIS. The remaining sample was allocated proportionally to the noncertainty industry strata based on payroll.

For the remaining strata, a simple random sample was selected without replacement. BRDI-M coverage was intended to complement BRDIS coverage for the 1–4 employer component of the business sector, but there is also overlap with the 5–9 employee portion. The BRDIS sample was selected first and the BRDI-M sample was selected from those cases not selected in the BRDIS sample.

In the large stratum, probabilities of selection were assigned based on the proportion of the industry total on the large stratum frame. In the small stratum, probabilities of selection were assigned by using proportions of state and industry totals on the frame. The exception was for companies with an “Other nonmanufacturing” (ONM) NAICS code that had their probabilities set so the maximum weight would be approximately 100. The stratum of small companies was the primary focus of the survey and received most of the sample cases. The selection probabilities of the BRDI-M sampling units were adjusted to account for the BRDIS companies in the frame that were not removed from the sampling frame.

Once selected, each company was assigned a sampling weight equal to the reciprocal of its probability of selection for the sample. Companies that were selected for the sample with certainty were assigned sampling weights equal to 1, companies that were selected in an industry other than ONM had a maximum weight of 23, and companies that were selected in an ONM industry had a maximum weight of 100. A complete and detailed description of the sample design and estimation methodology is given in the 2016 BRDI-M methodology report available from the NCSES project officer.

With the above sample design parameters, a total of 200,000 companies were selected. The small stratum contained 161,066 companies (86,939 certainties), and the large stratum contained 38,934 companies (15,482 certainties). After sample selection, 9 companies no longer had a domestic address on the Business Register. These companies were dropped from the sample, leaving a final sample size of 199,991 companies.

In addition to paper questionnaires, an electronic mode of data reporting via the U.S. Census Bureau’s Centurion data collection instrument was available to all BRDI-M respondents. Respondents were made aware of Centurion in BRDI-M-related correspondence and transmittals from the U.S. Census Bureau. For paper versus electronic response rates, see section Response by Mode.

For the 2016 cycle of BRDI-M, a single questionnaire was used to collect data for the survey.

Of the companies surveyed for the 2016 survey, 22.2% did not submit any response, and an additional 0.5% did not provide enough information to be treated as responses. Nonresponse studies are conducted periodically to assess reasons for nonresponse and possible nonresponse bias. Two metrics used by NSF and the U.S. Census Bureau to measure unit response to BRDI-M were check-in rates and unit response rates.

Check-in rate. The check-in rate is defined as the unweighted number of surveys that were either mailed in or submitted online by in-scope companies, divided by the unweighted total number of all in-scope companies in the sample. Response to individual questions did not factor into this metric.

Unit response rate (URR). The URR is the unweighted number of responding companies with positive data for at least one of the survey’s key items (i.e., another U.S. company owned more than 50%, ceased operations, or employment), divided by the unweighted total number of in-scope companies in the sample.

For companies with 1–4 employees, the check-in rate was 77.2%, and the URR was 76.7%.

BRDI-M collects data for over 40 variables, and the distribution of values reported by sample companies is highly skewed. Thus, rather than report unweighted item response rates, total quantity response rates are calculated, which are based on weighted data.

Total quantity response rate (TQRR). For a given published estimate other than count or ratio estimates, TQRR is the percentage of the weighted estimate based on data that were reported by units in the sample or on data that were obtained from other sources and were determined to be equivalent in quality to reported data. The TQRR for total R&D performed in the United States for companies with 1–4 employees in 2016 was 76.4%.

Total quantity nonresponse rate (TQNR). For a given published estimate, TQNR, defined as 100% minus TQRR, is calculated for each tabulation cell from BRDI-M, except for cells that contain count or ratio estimates. TQNR measures the combined effect of the procedures used to handle unit and item nonresponse on the weighted BRDIS estimate. TQNR tables corresponding to each data table are available from the NCSES project officer.

Overall, 42% of checked-in cases responded to BRDI-M by mailing in the paper form, and 58% responded using the online version of the survey.

Given the size and complexity of BRDI-M, many survey responses included errors that required correction or unusual patterns that required validation. Several edit checks were programmed to improve the efficiency of analyst data review and correction (see ABS Methodology Report, Appendix A, available from the project officer).

BRDI-M had 20 separate edits to check for outliers. Of those 20 edits, 13 were “range tests,” 5 were “survey rule tests,” and 2 were “balance tests.” The edit checks were designed to flag outliers for further analyst review (analytical edits). Descriptions of the data edits and edit failure rates are in annual methodology reports available from the NCSES project officer.

During the editing and review process, several cases were flagged as potential false-positive cases. The analysis of false-positive cases began by using a prepared data set with the following data items: company ID, status, check-in date, sample weight, company name, contact e-mail address, sample NAICS, primary business, employment variables, sales, R&D expenditures, and R&D yes-or-no indicators. Variables for the companies’ 2-, 3-, and 4-digit NAICS were added to the data set as well as categorical variables for R&D activity (1 = positive R&D, 0 = no R&D), and R&D intensity or R&D-to-sales ratio (R&D/sales). Once the data set was complete, SAS, JMP, and Excel was utilized to calculate the following for each NAICS level: count of sampled companies, count of active companies, checked-in cases, and count of cases with positive R&D, median R&D expenditures, and mean R&D expenditures. These summarized data were then used to identify possible outliers by sorting the data by descending R&D and NAICS level. Once the data were sorted by largest R&D, BRDI-M analysts analyzed the following:

• Are units reasonable? If not, flag for correction.
• Does reported primary business align with sample NAICS? If not, flag for further review.
• Does company explicitly note R&D activities in its primary business? If so, OK. If not, check company website (if available) for evidence of R&D.
• If the R&D-to-sales ratio is high, is it likely that the company is reporting all business expenses as R&D? This is only valid if the company is a startup or is doing R&D that is paid for by a customer, grant, or business partner.

In addition, BRDI-M analysts also reviewed R&D cases in NAICS levels where R&D is rare. For statistical purposes, R&D involves creative, systematic work aimed at resolving scientific or technological uncertainty. Respondents, however, sometimes apply a more common definition of research or development that includes activities instructed to be excluded, such as market research, literature reviews (“researching” current knowledge from published papers, books, and other resources), routine testing or technical activities (including routine software development and website design), and management or policy studies. When reviewing the R&D cases in NAICS levels where R&D is rare, the BRDI-M analysts looked for the following:

• Does reported primary business align with sample NAICS? If not, flag for further review.
• Does company explicitly note R&D activities in its primary business? If so, OK. If not, check company website (if available) for evidence of R&D.
• Is there any indication on company website that this case is more likely to have R&D activity than its peers? If not, flag as possible false positive. This involves analyst judgement.

For various reasons, many firms chose to return the survey questionnaire with one or more blank items. For some firms, internal accounting systems and procedures may not have allowed quantification of specific expenditures. Others may have refused to answer any questions as a matter of company policy. Weighted estimates produced from BRDI-M include adjustments to account for companies that did not respond to the survey (unit nonresponse) and for companies that did respond but left some questions blank (item nonresponse).

Unit nonresponse is handled by adjusting weighted reported data and imputed data as follows. Each company’s sampling weight is multiplied by a nonresponse adjustment factor. To calculate the adjustment factors, each company in the sample that is eligible for tabulation is assigned to one (and only one) adjustment cell. The adjustment cells are based on the industry-based strata for sampling described in the section Stratification of the Sample Frame. For a given adjustment cell, the nonresponse adjustment factor is the ratio of the sum of the weights for all companies in the cell to the sum of the weights for all companies in the cell with reported or imputed data.

Item nonresponse for a given company is handled by item imputation. Data are imputed by programmed item imputation procedures. Depending on the particular item being imputed for a company, these procedures are based on a combination of (1) direct substitution of available company data, and (2) imputation using averages from other companies in the industry. Tables of imputation rates corresponding to each data table are available from the NCSES project officer.

The general methodology used to produce estimates from BRDI-M involves sums of weighted data (reported or imputed) in which the weights are the product of the sampling weight and the nonresponse adjustment factor.

Estimates published for BRDI-M are computed as sums of weighted data for sample companies that reported to the survey or for sample companies for which data could be reliably imputed based on prior reports or other information. Two types of weights are used for estimates of R&D: sampling weights, and nonresponse adjustment factors. The sampling weight for a given company is calculated as the reciprocal of the company’s probability of inclusion in the sample. Nonresponse adjustment factors are used to represent companies in the sample that did not provide sufficient response data to be directly tabulated and whose data could not be imputed. For information on the calculation of the nonresponse adjustment factors, see section Unit Nonresponse.

Each value that contributes to a given BRDI-M estimate is multiplied by both its sampling weight and its nonresponse adjustment factor (if applicable), and these weighted values are then summed to create the estimate.

The estimates produced from BRDI-M are subject to both sampling and nonsampling errors.

Potential nonsampling errors include coverage error and various response and operational errors, such as errors during data collection, reporting errors, transcription errors, and bias due to nonresponse. These are all types of errors that could also occur if a complete enumeration of the sample frame had been conducted under the same conditions as the sample survey. Most of the important operational errors were detected and corrected during the course of reviewing data for reasonableness and consistency. Though nonsampling error is not measured directly, quality control procedures were employed throughout the survey process to minimize this type of error.

Sampling error is the difference between estimates obtained from the sample and results theoretically obtainable from a comparable complete enumeration of the sample frame. This error results because only a subset of the sample frame is measured in a sample survey. For published estimates from BRDI-M, standard errors are produced for estimated percentages, while relative standard errors (RSEs) are produced for all other estimates. Tables of the estimated measures of sampling variability corresponding to each data table are available from the NCSES project officer.

Standard errors may be used to define confidence intervals about the corresponding estimates with a desired level of confidence. If a confidence interval were constructed for each possible sample that could be selected, then it would be expected that the percentage of confidence intervals containing the result of a complete enumeration of the sample frame would equal the percentage of the level of confidence. For example, the interval defined by a margin of error of two standard errors yields a confidence interval of approximately 95%.

Because relatively few companies perform R&D in the United States, and because the amount of R&D they perform is quite variable, it is difficult to achieve control over the sampling error of survey estimates produced from BRDI-M. The sample size is sufficiently large that estimates based on the total sample are subject to low sampling error. However, because priority in designing the sample was given to industries that were expected to have a higher percentage of companies with R&D expenditures, the sampling error may be larger for estimates for the lower-priority industries. The RSE for the estimate of total domestic R&D performed by all microbusinesses was 3.74% in 2016.

Variations in respondent interpretations of the definitions of R&D activities are of particular concern. Little public information exists for most of the small businesses surveyed by BRDI-M, so it is difficult to determine whether or not companies are reporting R&D that satisfies the survey’s definitions, particularly where the development of software and internet applications are concerned. While no metric of measurement error is produced, ongoing efforts to minimize measurement error include questionnaire pretesting, improvement of questionnaire wording and format, inclusion of more cues and examples in the questionnaire instructions, in-person and telephone interviews and consultations with respondents, and post-survey evaluations.

Employment, total and R&D. Involves the number of people employed by the company as well as those involved in R&D activities during the pay period that included 12 March of the survey year (2016). (The date 12 March is what most employers use when paying first-quarter employment taxes to the Internal Revenue Service.) R&D employees are those who provide direct support to R&D, such as researchers, R&D managers, technicians, clerical staff, and others assigned to R&D groups. Those not included are employees who provide indirect support to R&D, such as corporate personnel, security guards, and cafeteria workers.

Innovation. BRDI-M questions on innovation activities refer to product, process, marketing, and organizational innovation. A product innovation is the market introduction of a new or significantly improved good or service with respect to its capabilities, user-friendliness, components, or subsystems. A process innovation is the implementation of a new or significantly improved production process or delivery method for the company’s goods or services. Product and process innovations (new or improved) must be new to the respondent company, but they do not need to be new to the company’s market, and the innovations could have been originally developed by the respondent company or by other companies. A marketing innovation is the implementation of a new marketing method involving significant changes in product design or packaging, product placement, product promotion, or pricing. An organizational innovation is the implementation of a new organizational method in the company’s business practices, workplace organization, or external relations.

R&D and business R&D. R&D is planned, creative work aimed at discovering new knowledge or developing new or significantly improved goods and services. This includes (1) activities aimed at acquiring new knowledge or understanding without specific immediate commercial applications or uses (basic research), (2) activities aimed at solving a specific problem or meeting a specific commercial objective (applied research), and (3) systematic use of research and practical experience to produce new or significantly improved goods, services, or processes (development). R&D includes both direct costs, such as salaries of researchers, and administrative and overhead costs clearly associated with the company’s R&D. However, R&D does not include expenditures for routine product testing, quality control, and technical services unless they are an integral part of an R&D project. R&D also does not include market research; efficiency surveys or management studies; literary, artistic, or historical projects, such as films, music, or books and other publications; and prospecting or exploration for natural resources.

Sales. Dollar values for sales, revenues, and grants. Included are revenues from the company’s domestic operations. If a respondent company is owned by a foreign parent company, sales to the parent company and to affiliates not owned by the respondent companies are included. Excluded are nonoperating income such as dividends and interest as well as excise, sales, and other revenue-based taxes.

The data from BRDI-M can be found online at https://www.nsf.gov/statistics/srvymicrobus/#tabs-2. Information from BRDI-M is also included in Science and Engineering Indicators and in National Patterns of R&D Resources.

BRDI-M contains confidential data that are protected under Title 13 and Title 26 of the United States Code. Two types of data are currently available: public-use tabular statistics, and restricted microdata. Detailed tabular statistics can be obtained by contacting the BRDI-M project officer. Microdata for the BRDI-M can only be accessed at the U.S. Census Bureau’s secure Research Data Centers (RDCs). To learn more about RDCs and for instructions on how to apply for data use, please visit the Center for Economic Studies page on research opportunities.