Purpose. The National Training, Education, and Workforce Survey (NTEWS) Pilot provides data on the educational and training characteristics of the nation’s workforce, with a focus on those in the skilled technical workforce. The NTEWS Pilot is a comprehensive source of information regarding the education and work credentials held by the adult U.S. population.

The estimates included in the NTEWS Pilot data tables are not official statistics and should not be used to make official statements or inferences about characteristics of the population or economy.

The NTEWS is in a pilot phase. Survey methods including data collection and post-collection processing are being evaluated, and comprehensive quality measures are not available. The 2022 NTEWS Pilot data tables are designated as an experimental statistical product. National Center for Science and Engineering Statistics (NCSES) experimental statistical products are created to benefit users in the absence of other relevant information and are developed using innovative and exploratory methodologies. The NTEWS Pilot data tables are published to engage data users and other stakeholders in the survey’s development to improve quality for future iterations of the survey. Experimental statistical products may not meet some of NCSES’s quality standards and, as a result, users should assess the utility limitations of these experimental statistics relative to the intended use. More information about the NCSES quality standards is available on the NCSES quality and transparency website.

The NTEWS Pilot focuses on two areas of national interest. First, the NTEWS Pilot provides a unique set of data on the characteristics of the nation's adult population, particularly the skilled technical workforce—individuals in science, technology, engineering, and mathematics (STEM) occupations that require a high level of knowledge in a technical domain but do not require a bachelor’s degree for entry. Second, the survey also gathers data on adults’ non-degree credentials (certifications, licenses, postsecondary certificates, and work experience programs [WEPs]) and employment characteristics, to understand the prevalence of work-related credentials and the relationship between work-related credentials and employment outcomes.

The National Survey of College Graduates (NSCG) complements these data with information on individuals residing in the United States with at least a bachelor’s degree, including those who received degrees only from foreign institutions.

The Survey of Doctorate Recipients (SDR) further complements these data with information on the population of U.S.-degreed doctoral-level scientists and engineers.

The NTEWS Pilot, NSCG, and SDR are sources of detailed information that support a broad range of policy and research topics on the dynamics of the educated and trained workforce over time. Historically, NCSES has conducted the NSCG and SDR surveys every two to three years and provides both cross-sectional and longitudinal data on the education and employment of the college-educated U.S. science and engineering (S&E) workforce. This NTEWS Pilot data collection effort was the first cycle for providing data for the skilled technical workforce population.

Data collection authority. The information collected in the NTEWS Pilot is solicited under the authority of the National Science Foundation Act of 1950, as amended; the America COMPETES Reauthorization Act of 2010; and the Education Sciences Reform Act of 2002. The Census Bureau collects the NTEWS Pilot data under the authority of Title 13, Section 8 of the United States Code. The Office of Management and Budget (OMB) control number is 3145-0264. The disclosure review number is NCSES-DRN24-050T.

Survey contractor. The Census Bureau, under NSF interagency agreement, collected, processed, and tabulated the statistics in this report.

Survey sponsor. National Center for Science and Engineering Statistics (NCSES) within the U.S. National Science Foundation (NSF) and the National Center for Education Statistics (NCES) within the Department of Education.

Frequency. Periodic, to be determined.

Initial survey year. 2022.

Reference period. April 2022 to October 2022.

Response unit. Individuals ages 16 through 75 who are not enrolled in high school.

Sample or census. Sample.

Population size. Approximately 249.8 million individuals.

Sample size. Approximately 43,000 individuals.

Target population. The NTEWS Pilot target population includes individuals who meet the following criteria:

• Reside in the United States or Puerto Rico according to the 2018 American Community Survey (ACS) interview
• Not institutionalized according to the 2018 ACS interview
• Between the ages of 16 and 75 (inclusive) as of 1 March 2022
• Not enrolled in high school at the time of their NTEWS Pilot interview

Sampling frame. The Census Bureau’s 2018 ACS was used to create the sampling frame. The ACS samples about 3 million households on a yearly basis to ask about jobs and occupations, educational attainment, veteran status, whether people own or rent their homes, demographics, and other topics. The NTEWS Pilot frame included all individuals from the ACS who met the following eligibility requirements:

• Resided in the United States, including the District of Columbia and Puerto Rico, as of the ACS interview date
• Not institutionalized as of the ACS interview date
• Ages 16 through 76 as of 1 March 2022
• Did not have blank or generic name information, and did not have incomplete address information (i.e., address not fully blank or not missing state plus at least one of the following: street/address line, city, or zip code) on the ACS data file

To be eligible for the NTEWS Pilot, a sample person must meet all the above criteria plus not be enrolled in a primary or secondary school (K-12). This eligibility criterion was added to the NTEWS Pilot questionnaire because this information could not be determined from the sampling frame data at the time of sample selection.

Sample design. The NTEWS Pilot is a cross-sectional study with a stratified sampling design. As a cross-sectional study, the NTEWS Pilot provides estimates of the size and characteristics of adults in the workforce for a specific calendar year. As part of the rotating panel design, every new panel receives a baseline survey interview and three follow-up interviews before rotating out of the survey.

The sample design uses a multiphase, stratified sample. The first step draws eligible respondents from the ACS sample. Then, the NTEWS Pilot sample persons are selected from the eligible ACS respondents using systematic probability proportional to size (PPS) sampling. The measure of size was defined as a person’s Name-and-Address adjusted weight, which is the final ACS simplified person-level weight adjusted for recent immigration undercoverage and removal of cases with incorrect contact information (e.g., bad names or incomplete addresses) from the preliminary sampling frame. Thus, a person with a higher Name-and-Address adjusted weight will have a higher probability of being selected into the sample. The formation of the sampling strata is based on the multiway cross of four stratification variables into 24 strata:

• Educational attainment (3 levels)
• Sex (2 levels)
• Underrepresented minority status (2 levels)
• STEM workforce status (2 levels)

  The NTEWS Pilot also oversampled adults who did not have a bachelor's degree and adults in the skilled technical workforce to improve the precision of estimates. Approximately 43,000 individuals were sampled from the 2018 ACS for the 2022 NTEWS Pilot.

Data collection. The data collection period for the 2022 NTEWS Pilot was 26 weeks (21 April 2022 to 24 October 2022), or approximately six months. The NTEWS Pilot used a trimodal data collection approach: a self-administered online survey (Web), a self-administered paper questionnaire (via mail), and a computer-assisted telephone interview (CATI). The survey was offered in English and Spanish. Several methods were used to contact sample persons to request they complete the survey. Sample persons were contacted through the following methods:

• By mail, with multiple invites and reminders sent to complete the survey online and some mailings including the paper questionnaire
• Through the CATI operation, which was also used for the nonresponse follow up (NRFU) stage
• By calling twice using an automatic dialer phone tree, with a reminder message to complete the survey

Throughout the collection, Census staff searched for new addresses and phone numbers for sample persons who received multiple undeliverable mailings or had unproductive phone numbers.

Quality assurance procedures were in place to monitor key data collection activities to ensure operations progressed in a timely manner and were performed according to plan. These activities included printing, mail package assembly, mailout, questionnaire check-in, data keying, coding, telephone questionnaire assistance (TQA), scanning questionnaires into the document database, coding, and post-data collection processing.

Mode. About 57% of respondents completed the survey by Web, 33% by mail, 8% by CATI, and 2% by TQA. Of those who completed by Web, 65% were on a computer, 32% were on a smartphone, and 3% were on a tablet.

Each of the three modes of data collection (Web, mail, or CATI) was also translated into Spanish for sample persons to use while responding. About 15% of all CATI responses were conducted in Spanish, 7% of all mail responses used the Spanish questionnaire, and 3% of all web responses used at least one Spanish question screen.

Response rates. Response rates were calculated on complete responses from instruments. To be considered a complete response, the following critical items must have been answered.

• Working for pay or profit
• Looking for work
• The name of the main job or the description of the main job
• Educational attainment
• Current high school enrollment
• Birthdate (to determine age)
• Living in the U.S.

The unweighted and weighted response rates adjusted for estimated ineligible individuals for 2022 NTEWS Pilot were 39% and 44%, respectively. Of the roughly 43,000 sample persons, approximately 15,500 completed the survey.

Data editing. Response data had initial editing rules applied relative to the specific mode of capture to check internal consistency and valid range of response. The Web survey captured most survey responses and had internal editing controls where appropriate. The Integrated Computer Assisted Data Entry (iCADE) system processed the mailed paper surveys. Responses from the three modes were merged for subsequent coding, editing, and cleaning necessary to create an analytical database.

Coding for the NTEWS Pilot survey data took open-text, verbatim answers and converted them to a standardized code. Separate coding operations corresponding to survey questions were used: field of study (i.e., NCES Classification of Instructional Programs), credential (certifications and licenses), other-specify (e.g., other reason why a respondent earned a degree or certificate, how often a respondent with a credential needs to renew it, and name of the organization or agency that issued the credential), work experience program, industry, and occupation. Each operation focused on a specific set of variables to be coded from the survey questions.

Imputation. Missing items except for critical items were imputed. Imputation is performed for several reasons. Many data users prefer data files without missing data, especially those unfamiliar with techniques used to analyze missing data. In addition, some statistical packages cannot process an observation with missing variable responses. Eliminating missing data can also reduce nonresponse bias. However, imputation can affect the variance estimates and possibly introduce other bias. For this reason, users have the option not to use imputed data because imputed values are flagged in the data file. Users can request the imputation-flag file from the Survey Manager.

The NTEWS Pilot used a combination of logical and statistical imputation. During the editing process, logical imputation was used to determine the answer to one question with missing data based on the answer to another related question when feasible. In some circumstances, edit checks found inconsistent data, which were removed and then subjected to statistical imputation through the hot deck imputation procedure, which involves splitting individuals into similar cells using class variables. The class variables may include employment status or education level, etc. Within a particular class, all observations are sorted using a list of sort variables, such as number of hours worked per week or age group. Then, an observation with a missing value is given the same response as the nearby observation not missing a value, called the donor, in the order of sort. The purpose is to find a donor who is most like the respondent and most likely to respond the same. The donor is sometimes referred to as the nearest neighbor.

Class and sort variables were selected so that donors and recipients would be as alike as possible in relation to the variable being imputed. Class variables were chosen primarily from filter variables for the questionnaire skip patterns. Sort variables were selected using stepwise-variable regression models to determine significant predictors for the item to be imputed. Potential class and sort variables came from responses to the 2018 ACS or from the NTEWS Pilot items that were already imputed or never missing, along with paradata and recodes of ACS and NTEWS Pilot variables. Sort variables were primarily listed in the order of their significance in the regression model. However, some variables were demoted down the sort order, which prevents these variables from dominating the sort in the list and helps to maintain consistency.

The item nonresponse rates reflect data missing after logical imputation or editing but before statistical imputation. These rates provide information on the percentage of data that was statistically imputed for each questionnaire item. For key variables—such as employment status, current employment educational background, licenses and certifications, and certificates—the weighted item nonresponse rates ranged from 2.5% to 10.2%. Nonresponse to questions deemed sensitive was higher: nonresponse for the type of visa of non-U.S. citizens was 20.6% and for earnings was 10.9%. Nonresponse was also higher for information that may be difficult to recall, such as the last month and year worked for those who were not employed, which was 17.3 % and 12.1%, respectively. Imputation rates frequently varied by educational attainment—generally, variables demonstrated higher imputation rates for lower education levels (i.e., individuals with a high school diploma or less).

Imputation was not performed on critical items or verbatim-based variables. For some missing demographic information, the NTEWS Pilot imported the corresponding data from the ACS, which had performed its imputation.

Weighting. Sampling weights were created for each respondent to support population estimates because the NTEWS Pilot is based on a complex sampling design and is subject to nonresponse. The final analysis weights account for several factors, including the following:

• Adjustments to account for undercoverage for recent immigrants
• Adjustment for incorrect names or incomplete address information on the sampling frame
• Unequal sample selection probabilities to produce base weights
• Post-stratification adjustment to control back to the frame totals within cells defined by cross of sampling cell (Hispanic origin, non-Hispanic White/Other race flag, disability status, age group, and occupation group) with small cells collapsed
• Adjustment to account for the removal of duplicate cases
• Adjustment to account for non-locatability and unit nonresponse during data collection
• Trimming for extreme weights
• Raking adjustments to reallocate the trimmed weights back to the pre-trim totals
• Post-stratification adjustments that ratio adjusts the weights to ensure consistency with the population control totals for key demographic characteristics

The final analysis weights enable data users to derive survey-based estimates of the NTEWS Pilot target population. The variable name on the NTEWS Pilot public use data file for the NTEWS Pilot final analysis weight is FSNTW2201.

Variance estimation. The successive difference replication method (SDRM) was used for variance estimation. The theoretical basis for the SDRM is described in Wolter (1984) and in Fay and Train (1995). As with any replication method, successive difference replication involves constructing numerous subsamples (replicates) from the full sample and computing the statistic of interest for each replicate. The NTEWS Pilot used 80 replicates; each replicate underwent the same weighting adjustment as the full sample.

Disclosure protection. The estimates presented in the NTEWS Pilot data tables are rounded to the nearest 1,000 to protect against disclosure of confidential information provided by NTEWS Pilot respondents.

Data table cell values based on counts of respondents that fall below a predetermined threshold are deemed sensitive to potential disclosure, and the letter “D” indicates this type of suppression in a table cell.

Sampling error. NTEWS Pilot estimates are subject to sampling errors. Estimates of sampling errors associated with this survey are calculated using replicate weights. Data table estimates with coefficients of variation (i.e., the estimate divided by the standard error) that exceed 50% are deemed unreliable and are suppressed. The letter “S” indicates this type of suppression in a table cell.

Coverage error. Coverage error occurs in sample estimates when the sampling frame does not accurately represent the target population of interest. Because the 2022 NTEWS Pilot used the 2018 ACS as the originating sampling frame, differences in the population from 2018 to 2022 result in some coverage error, and any potential undercoverage present in the ACS transferred to the 2022 NTEWS Pilot design. Additional undercoverage errors may exist because of self-reporting errors in the NTEWS Pilot sampling frame that led to incorrect determination of survey eligibility of individuals.

Nonresponse error. The weighted response rate for the 2022 NTEWS Pilot was 44%, and the unweighted response rate was 39%. A preliminary nonresponse bias analysis study for the 2022 NTEWS Pilot concluded the potential for bias was minimal among certain large domains, but there is the potential for greater nonresponse bias across other domains with less robust response. Analysis of NTEWS Pilot nonresponse trends was used to develop nonresponse weighting adjustments to minimize the potential for nonresponse bias in the NTEWS Pilot estimates. A hot deck imputation method was used to adjust for item nonresponse.

Measurement error. The NTEWS Pilot is subject to reporting errors from differences in the interpretation of questions and by mobility (Web, mail, or CATI). To reduce measurement errors, the NTEWS Pilot questionnaire items were pretested in two independently conducted cognitive research in 2018 and 2021.

The administration of the 2022 NTEWS Pilot is the first cycle of the survey. The NTEWS Pilot targets the same population as the Adult Training and Education Survey (ATES) conducted by NCES (i.e., adults in the United States who are not in enrolled in high school). The NTEWS Pilot has overlapping populations with the NSCG because both survey collections use the ACS as their sampling frame. Also, many NTEWS Pilot questions use similar survey questions that were used on the ATES or NSCG.

There are two data files available to users. The NTEWS Pilot public use file (PUF), which removed all personally identifiable information (PII), is available as a downloadable file through the NCSES data page (https://ncses.nsf.gov/explore-data). The NTEWS Pilot restricted-use file (RUF) is available through the Census Bureau’s Federal Statistical Research Data Centers (https://www.census.gov/fsrdc).

Changes in survey coverage and population. Not applicable.

Changes in questionnaire. Not applicable.

Changes in reporting procedures or classification. Not applicable.

Disability. The NTEWS Pilot asks the degree of difficulty—none, slight, moderate, severe, or unable to do—an individual has in seeing (with glasses), hearing (with a hearing aid), walking without assistance, lifting 10 pounds, or concentrating, remembering, or making decisions. Those respondents who answered “moderate,” “severe,” or “unable to do” for an activity were classified as having a disability.

Earnings. Wages, salary, commissions, bonuses, or tips from the individual’s main job during the previous 12 months. This amount is before deductions for taxes, bonds, dues, or other items.

Race and ethnicity. Ethnicity is defined as Hispanic or Latino or not Hispanic or Latino. Hispanic or Latino may be any race; race categories exclude Hispanic origin. Values for those selecting a single race are American Indian or Alaska Native, Asian, Black or African American, Native Hawaiian or Other Pacific Islander, and White. Respondents who indicate they are not Hispanic or Latino and select two or more races are reported as “More than one race.”

Skilled technical workforce. The skilled technical workforce comprises workers in STEM occupations (S&E, S&E-related, and middle-skills occupations) who do not have an educational attainment of a bachelor's degree or higher. See the NCSES skilled technical workforce website for more information (https://ncses.nsf.gov/initiatives/projects-partnerships/skilled-technical-workforce).

Underrepresented minority. Demographic groups that are underrepresented in science and engineering, relative to their numbers in the U.S. population: American Indian or Alaska Native, Black or African American, and Hispanic or Latino. For detailed data on racial and ethnic representation, see the 2023 NCSES report Diversity and STEM: Women, Minorities, and Persons with Disabilities 2023. 

Fay RE, Train GF. 1995. Aspects of Survey and Model-Based Postcensal Estimation of Income and Poverty Characteristics for States and Counties. American Statistical Association Proceedings of the Section on Government Statistics, 154–59.

Wolter K. 1984. An Investigation of Some Estimators of Variance for Systematic Sampling. Journal of the American Statistical Association 79(388): 781–90.