Using Mechanical Turk in Study of Individual Innovation

NCSES 22-201 | March 31, 2022

| Jesse Chandler, Audrey Kindlon and Rebecca Morrison

Disclaimer

Working papers are intended to report exploratory results of research and analysis undertaken by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF). Any opinions, findings, conclusions, or recommendations expressed in this working paper do not necessarily reflect the views of NSF. This working paper has been released to inform interested parties of ongoing research or activities and to encourage further discussion of the topic.

This working paper represents the work of the authors and is not considered official government statistics.

Abstract

Innovation is typically studied within businesses, but other sectors of the economy, such as governments and the household sector, can also undertake innovation activity. In order to fully understand innovation at the economy and society level, innovation must be understood beyond the business sector. However, there are numerous challenges to understanding non-business innovation. Individual innovation in particular is assumed to be relatively rare in the general population and thus expensive to measure using probability-based samples. At the same time, relatively few studies have examined individual innovation using non-probability samples. The lack of research on individual innovation leads to uncertainty about the best ways of asking questions related to individual innovation or which topics to prioritize if a probability-based sample is ever used to estimate individual innovation rates. In 2019, NCSES conducted a study of innovation at the individual level using Amazon Mechanical Turk (MTurk) for recruiting respondents and administrating the survey.

Introduction

This working paper presents an examination of individual innovation using a crowdsourcing tool for survey administration. Innovation is typically studied within businesses, but other sectors of the economy, such as governments and the household sector, can also undertake innovation activity. Individuals in particular have recently been recognized as an important source of innovation.^{Organisation for Economic Co-operation and Development (OECD) and Statistical Office of the European Communities (Eurostat). 2018. Oslo Manual: Guidelines for Collecting, Reporting and Using Data on Innovation. 4th ed., p. 20. Paris.}

Individual innovation is important because sectors of the economy are porous, with innovations sometimes flowing from one sector to another (as is the case of a product conceived in a dorm room or garage launching a multinational company). In order to fully understand innovation at the economy and society level, innovation must be understood beyond the business sector.

In 2019, NCSES conducted a study of individual innovation using Amazon Mechanical Turk (MTurk). MTurk is an online labor market that matches “requesters” with “workers” who are willing to complete requested online tasks for payment.

The study was conducted in two phases, both conducted using MTurk. The first phase examined whether people can successfully apply a definition of individual innovation to a hypothetical vignette. The second phase was a survey of the prevalence of individual innovation and the characteristics of individual innovators in a sample of U.S. respondents. The second phase also used some of the vignettes developed in phase I to help participants understand our definition of innovation. The study examined whether the innovation activity reported by a self-selected sample was similar to that observed in nationally representative samples (for example, von Hippel, Ogawa, and de Jong 2011), thus indicating whether MTurk is a suitable platform to use for developing or testing surveys about individual innovation. The data and analysis presented in this working paper should not be considered representative of a specific population of individuals. Individuals who do not participate in MTurk or who otherwise would not participate in this study may have different views and experiences than the participants in this study.

What Is MTurk?

Amazon Mechanical Turk (MTurk) is an online labor market that matches “requesters” who have Human Intelligence Tasks (HITs) they wish to complete with “workers” who are willing to complete them for payment. MTurk is a widely used source of respondents in academic research (Anderson et al. 2019; Chandler and Shapiro 2016; Stewart, Chandler, and Paolacci, 2015). MTurk has a relatively large and diverse user base (Casey et al. 2017) that enables the fast, cost-effective, and flexible collection of data. Hundreds of respondents can be recruited per day for approximately 12 cents per minute, with no minimum study length or sample size. MTurk also provides technical features to support large-scale data collection, such as automated compensation for respondents and a reputation system that allows requesters to avoid workers with a history of providing low-quality responses.

Although MTurk will not produce a representative sample of respondents, it does offer a diverse sample that could resemble the actual population enough to make it a useful tool for pilot-testing survey instruments. The scale of the platform and a variety of respondents available for pilot testing represents a substantial improvement over samples obtained by current (typically in-person) data collection methods that are constrained by research respondents’ need to travel to the cognitive testing site (Edgar, Murphy, and Keating 2016). To the extent that associations between variables within nonrepresentative samples approximate associations between variables within representative samples, MTurk may also be an inexpensive way to explore and prioritize hypotheses for study using nationally representative samples.

Defining Individual Innovation

The Organisation for Economic Co-operation and Development (OECD) Oslo Manual (2018) provides a general definition of an innovation as “a new or improved product or process that differs significantly from a unit’s previous products or processes and that has been made available to potential users (products) or brought into use by the unit (processes).”^{Organisation for Economic Co-operation and Development (OECD) and Statistical Office of the European Communities (Eurostat). 2018. Oslo Manual: Guidelines for Collecting, Reporting and Using Data on Innovation. 4th ed., p. 20. Paris.} Products can refer to both goods and services, whereas processes refer to the policies, production, marketing, and delivery techniques that are used to produce a product.

Individuals in particular have recently been recognized as an important source of innovation. One author calls it the democratizing of innovation.^{von Hippel E. “Democratizing Innovation: The evolving phenomenon of user innovation.” An excerpt from the book Democratizing Innovation published by MIT Press. Available at https://www.oecd.org/sti/inno/37450155.pdf.} Individual innovations are defined slightly differently than business innovations. An important distinction is that firms can innovate not only through internal activities (like research and development) but also through acquisition by sale, licensing, or other arrangements. In contrast, individual innovation is focused more narrowly on innovations created by the individual. Thus, acquiring an innovative product or process is not considered an individual innovation.

Another distinction is that whether something is a business innovation depends largely on whether it is used by the firm or has been introduced on the market. Individuals engage in both production and consumption, including the consumption of goods they produce themselves. For this reason, retooling a factory to make products faster is a process innovation. In contrast, if an individual develops a novel jig to make a product faster, it could be regarded as creating a product (the jig), a process (integrating the jig into the production of the product), or both.

The OECD identifies two approaches to studying business innovation. One is to use firms as the unit of measurement (the “subject approach”) and focus research on the barriers and facilitators of innovative behaviors and their downstream economic impacts. The second is to use innovations as the unit of measurement (the “object approach”) and focus on the details of a single innovation. In this study, we adopt a mixture of approaches: we collect some information about innovators but focus primarily on validating whether a specific product is an innovation.

A product or process does not need to be particularly useful or unique to be classified as an innovation. A minority of innovations are radically different from anything seen before in the world, but most represent new or improved firm or market novelties or the diffusion of existing products to new industries or markets rather than disruptive innovations that are being introduced to the word for the first time. To be an innovation, a product must be used and must differ significantly from existing products in one or more objective properties (including the reduction of cost) or subjective properties (such as ease of use). Although the Oslo Manual (2018) distinguishes between improvements to the “look” of a product that enhance the user’s utility (innovations) and minor aesthetic changes (non-innovations), in the study presented here we generally do not classify purely aesthetic products as innovations. For individual innovations, cost is not always considered an attribute that defines a product as an innovation: products could be considered an innovation because they reduce other costs, but a production cost for an individual innovation that is lower than the purchase price of a commercially available alternative does not in and of itself indicate an innovation.

Phase 1 Pilot Study

Overview

For the first phase of the study, MTurk was used to test whether people could successfully apply a definition of individual innovation used by NCSES to classify whether everyday activities are innovative.

Methodology and Questionnaire

A sample of 303 U.S. residents was recruited from MTurk to complete an instrument developed by NCSES and paid $1.50 to complete a 20-minute survey (appendix A). All respondents (aside from one missing response) were residents of the United States, Puerto Rico, or another U.S. territory. The average age of the sample was 35, with a range of 19 to 71 years of age, and 41% of respondents were female. After reviewing a consent form and a definition of individual innovation, each respondent read seven descriptions of activities that involved the creation of six products (randomly selected from 18 possible innovations described in appendix B) and the creation of one process (randomly selected from four possible innovations). Participants could review the definition of innovation at any time during these tasks.

The definition of individual innovation was based on the one found in the Oslo Manual:

An innovation is a new or improved product (good or service) or process that a person(s) develops on their own time, is not work-related, and is used by the developer(s) or has been made available to other potential users. The innovation cannot be a home-built version of an existing product or process currently on the market. However, an innovation may include modifying existing products or processes to create something functionally novel.

To clarify the concept of “work-related”, a product or process is an innovation if someone does not create it for work. However, a person(s) can use their professional experience to create something in their leisure as long as it otherwise meets the definition above.

For each activity, there were two versions of each vignette. Participants were randomly assigned to either review a vignette that met the NCSES definition of individual innovation (and therefore was consistent with the OECD general definition of innovation) or a vignette that did not. The reasons for the vignette being non-innovation were the following:

Work related
Not used
Homemade version of something commercially available
Did not create something new

There are three dependent variables of interest: whether respondents can correctly classify vignettes as describing innovation, whether respondents review the definition of innovation before classifying a vignette, and how confident they are in their response (confidence was scored as 1, not at all confident; 2, somewhat confident; or 3, very confident). Finally, respondents reported their age, sex, and country of residence.

Findings

The study included three outcome variables for each vignette: (1) how the respondent classified the activity (coded as “correct” and “incorrect” or “innovative” and “not innovative,” depending on the specific research question addressed by an analysis), (2) whether respondents were very confident in their classification, and (3) whether respondents reviewed the definition of innovation while classifying the activity.

We found that respondents could usually recognize whether an activity was innovative. However, there was substantial variability in their responses and whether individual vignettes were correctly classified (table 1). Eight vignettes tested in phase 1 were classified as being high quality. For the purposes of this study, high quality is defined as 60% or more of respondents correctly classified the vignette as being innovative or non-innovative.

Table 1

Respondents who correctly classified activities in vignettes as innovative or non-innovative during the pilot study: 2019

(Percent)

^a Complete vignettes are listed in appendix A.

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Respondents correctly identified the innovative vignettes as innovative at least 66% of the time, regardless of the reason that a non-innovative activity was not innovative (table 2). Respondents also correctly identified the non-innovative vignettes as non-innovative more than one-half of the time, except for when the vignette was work related. Only 40% of respondents correctly identified work-related innovative vignettes as non-innovative. Items created at work may be innovative, but they are not considered individual innovations.

Table 2

Respondents who correctly classified activities in vignettes as innovative or non-innovative in the pilot study, according to the reason the non-innovative activity was not innovative: 2019

(Percent)

Note(s):

Each vignette had two versions—one version described an individual innovation and the other one did not. See appendix b for a complete listing of vignettes.

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Phase 2 Assessing the Prevalence and Characteristics of Innovators on MTurk

Overview

The second phase was an examination of the prevalence of individual innovation and the characteristics of individual innovators in a sample of U.S. respondents recruited from MTurk. The main purpose of this phase was to understand whether the incidence of innovation activity reported by a self-selected sample is similar to that observed in nationally representative samples (for example, von Hippel, Ogawa, and de Jong 2011).

Methods and Questionnaire

To explore the potential of MTurk as a tool for testing individual innovation surveys, we screened a sample of 9,654 U.S. residents from MTurk to find those who potentially created innovations on their own time and sought to have them complete an instrument developed by NCSES. These participants were paid 50 cents to complete a screener survey. The screener survey included a definition of individual innovation and three vignettes designed to check participant’s understanding of the definition (see appendix C). The vignettes were drawn from the Phase 1 sample of vignettes that proved most successful in terms of respondents accurately identifying the vignettes as innovation or not.

After reading a consent form, each respondent provided basic demographic information, including age, sex, race and ethnicity, state or territory of residence, and highest degree completed. The respondents were then given a definition of individual innovation supplemented by three vignettes designed to check their understanding. Respondents classified each vignette according to whether it was an innovation and received feedback about whether their classification was correct or incorrect and why. Respondents were then asked a series of questions to determine whether they created a product or artifact that met the NCSES definition of innovation. Specifically, we asked them whether (1) they created an individual innovation, (2) they did so at home or at work, (3) they created a new or modified product (and not just an imitation of an existing product), and (4) they or anyone else used this innovation.

We identified 638 people who potentially created an innovation (7% of the screened sample), 493 of whom agreed (77% of eligible cases) to complete a 20-minute survey that paid $3.00 for their time (table 3). The survey asked them to estimate the number of innovations they made across various product categories, answer detailed questions about a single (respondent-named) innovation, and give detailed demographic information.

Table 3

Results from the Individual Innovation Survey: 2019

(Percent)

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

This second survey captured more detail about respondents’ innovation activities. Respondents were asked whether they had created an individual innovation in each of eight product categories (software, household goods, transportation, tools, entertainment, education, care or medical products, and other products) during the 3-year period 2016–18, and if so, how many. Respondents who had innovations in multiple categories were asked to select the category that their most significant product belonged in, and all respondents were asked to describe their single most important innovation and what was new about it. Respondents then answered questions designed for us to assess whether this particular product or process was an innovation (that is, whether ready-made alternatives exist, whether they use the product themselves, and whether they created it for a business). Respondents also reported their motivation for creating the innovation, whether they collaborated with anyone to make it, and whether they took steps to protect, commercialize, or otherwise share their intellectual property. Finally, respondents provided additional demographic information, including whether they lived in an urban or rural area, the number of people who live in their household, the Washington Group Short Set of Disability Questions (2017), and their employment status, job category, and household income. The Phase 2 questionnaire is in appendix C.

Following the conclusion of data collection, the innovation descriptions were reviewed. For most products, it was assumed that they met the criteria for being an innovation unless there was clear evidence that they did not (following de Jong et al. ). However, for products that were primarily artistic or decorative in nature (for example, jewelry and household fixtures and furnishings) it was assumed they were not an innovation unless a novel function (even as simple as fitting in a specific amount of space) was described.

Findings

The survey included a definition of innovation, vignettes describing products to help clarify the definition of innovation, and screening questions about the nature of claimed innovations.

Findings compare favorably to other similar studies. The estimated incidence of individual innovation in this survey was 4.5% (table 3). To compare, Ogawa and Pongtanalert (2011) found a 5.2% individual innovation incidence rate for a representative sample of the U.S. population and a 3.7% individual innovation incidence rate for a representative sample in Japan.

It is possible to efficiently recruit moderately sized samples of innovators on MTurk. Workers were recruited by posting a Human Intelligence Task (HIT) on MTurk using TurkPrime, a third-party graphical user interface. To participate in the survey, workers had to have completed at least 100 assignments on MTurk and achieved a HIT Approval Ratio (HAR) greater than 95%. Workers with at least a 95% HAR were selected for this study because they produce higher-quality data (Peer, Vosgerau, and Acquisti 2014). Participation was further restricted to U.S. residents. Participants at IP addresses known to be registered to foreign countries or virtual private servers (U.S.-located data centers known to be used by foreign workers to circumvent IP address restrictions) were blocked from participating.

An advantage of MTurk is that participants can be rapidly screened for a fraction of the cost of a complete survey. Including participants who were screened out, an average of $17.29 in subject fees were paid for each completed survey.

Increased screening efficiently identifies individual innovation activity. The survey reported in this working paper used a more extensive screening method than that reported by Ogawa and Pongtanalert (2011, as reported by von Hippel, Ogawa, and de Jong 2011). Specifically, the Phase 2 survey included a definition of innovation, vignettes describing products to help clarify the definition of innovation, and screening questions about the nature of claimed innovations. The study conducted by Ogawa and Pongtanalert (2011) only asked respondents if they knew of an available equivalent product. As a result, substantially fewer respondents screened into the study presented here than into the study by Ogawa and Pongtanalert.

The difference in the incidence rates for these two studies suggests that much of the difference in screen-in rates seems to result from people correctly identifying that they are ineligible for the study. A related consideration is whether the screening procedure we used incorrectly screened out people who have created an innovation, and if so, how this could be avoided. For example, a more detailed description of process innovations may have helped more participants recognize and report process innovations they have created than found to be the case in the study presented in this working paper.

Together these findings suggest that screening processes may save time for potential respondents who would be screened out and would not have to respond to survey questions describing products that are not innovations. The screening process could also reduce the number of false-positive product descriptions that must be read by analysts.

Number and type of innovations. People who completed the survey reported producing an average of 6.5 innovations across 2.4 categories. Household goods (51%) and tools (46%) were the innovations reported most often, and care or medical products (14%) were the ones mentioned least often (table 4). There was some overlap between the kinds of categories people tended to have innovations.

Table 4

Categories for individual innovation and most significant innovation: 2019

(Percent)

Note(s):

Respondents can have innovations in multiple categories.

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Most significant innovations. Respondents who had innovations in multiple categories were asked to select the category that their most significant product belonged to. Using procedures similar to those used by de Jong (2016), the innovation was first validated by using a series of questions. Respondents were excluded from the analysis if they reported that the specific innovation could be substituted with a ready-made product, that it was never used, or that it was commercialized through their business or self-employment.

Twenty-eight percent of respondents reported that their most important innovation was computer related. One-fifth of the most important innovation was in tools (20%) or household goods (18%).

Most innovators on MTurk are not motivated by money. Most respondents who described a definitionally valid innovation were motivated to create it by personal need (90%), for fun (76%), or to develop skills (65%), while less than one-half of the respondents were motivated by a wish to help others (43%). Few respondents (14%) were motivated by money (figure 1). Only 16% of reported innovations were developed in collaboration with others, and one-half (52%) of these were developed in collaboration with one other person (table 5). Three out of five (58%) of these products required the innovator to spend money to develop them.

Figure 1

Motivation to create an existing innovation: 2019

Data View

(Percent)

Motivation	Percent
Personal need	89.9
Develop skills	64.5
Wish to help others	43.3
Make money	14.3
Fun	76.1

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Table 5

Collaboration during development of innovation, investment in innovation, and willingness to share innovation, among innovators: 2019

(Percent)

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Participants were generally interested in sharing their innovation with others: 39% said they have made the innovation available to others either free or at a price, whereas many others were either willing to freely share their innovation with anyone (22%) or selectively share it (30%); only 8% were unwilling to share their innovation with anyone.

Respondents generally did not attempt to protect their intellectual property: 88% of them reported that their innovation was not covered by patents, trademarks, copyright, trade secrets, or non-disclosure agreements (table 6). Few respondents were interested in trying to commercialize their innovation: 1% commercialized the innovation or intended to do so through an existing business, and none planned to start a new business to do so. Another 2% reported that a business bought or licensed their innovation with the intent to sell it, and 7% reported that they intended to contact a commercial business to adopt their innovation for sale.

Table 6

Intellectual property protection of innovation: 2019

(Percent)

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Innovators’ Characteristics

The median age of innovators was 39 years, just over one-half (52%) were male and over one-third (38%) were from the South, while roughly 20% each were from the Midwest (22%), the West (20%), and the Northeast (19%) (table 7). Innovators were well-educated, 58% had a 4-year degree or higher. Innovators were mostly White (84%). Approximately one in nine (11%) were African American, 8% each either were Asian American or were Hispanic or Latino, 3% were American Indian or Alaska Native, and 2% were Native Hawaiian or Pacific Islander.

Table 7

Demographic characteristics of innovators: 2019

(Percent)

Source(s):

National Center for Science and Engineering Statistics, Individual Innovation Survey, 2019.

Innovation behavior on MTurk is similar to innovation behavior in other samples. Patterns in the data observed in the study presented here parallel those observed in previous studies of innovation. This study found that innovators are more educated than non-innovators, which is consistent with differences observed by von Hippel and colleagues (2011; see also Miranda and Zolas 2018). This study also found that people reported creating new products and modifying old products with equal frequency (von Hippel, Ogawa, and de Jong 2011). Only about 16% of innovations were developed in collaboration with others, which is consistent with estimates of 10%–28% obtained from Ogawa and Pongtanalert (2011) and von Hippel and colleagues (2012); see also Miranda and Zolas 2018).

Similar to other studies (de Jong et al. 2015), this study found that individuals readily shared their innovations with others but generally did not commercialize them. However, it should be noted that the reported rate at which others adopted innovations for personal use in this survey (39%) was higher than that reported elsewhere (15%–20%: von Hippel, de Jong, and Flowers 2012; von Hippel, Ogawa, de Jong 2011). Finally, only 13% of respondents took steps to protect their intellectual property, which is again consistent with the rates observed in prior studies.

Respondents may struggle to count and classify innovations. There are correlations between innovations produced across different product categories, for example, between tools and household goods or between entertainment and education products. Although it is possible that these associations reflect the tendency for people to innovate across several categories, it is also possible that people count the same innovation in multiple categories. For example, an educational computer game could plausibly fall into the computers, entertainment, and education categories.

In some cases, respondents had difficulty understanding the categories that were provided. Products that should have been classified in a provided category (particularly household goods, tools, and self-care products) were sometimes classified as “other” innovations. This suggests that the definitions and examples for these categories that were provided might have been overly narrow. In particular, respondents did not seem to classify hygiene and beauty innovations as falling within care-related products.

Respondents can recognize innovations described in vignettes, but vignettes have an unknown impact on survey screening. In general, respondents were successfully able to classify the innovation vignettes in the screener questionnaire. Although the survey in Phase 2 differs from prior surveys of innovation on many dimensions—most notably the sample population, screener survey criteria, and coders reviewing respondent descriptions of innovations—it is possible that the feedback provided through answering questions about the vignettes helped respondents understand the definition of innovation.

Respondent’s ability to correctly classify products also seemed to predict whether they screened into the survey, although it is unclear why. One possibility is that respondents who paid attention to the definition were more likely to recall products that they created that met the criteria for an innovation. Another possibility is that these questions actually function like a data quality measure, identifying inattentive respondents.

Respondents are reluctant to share details of their innovations. Some participants were reluctant to disclose the nature of their innovations. Refusal rates for the survey seem high relative to our expectations for an MTurk sample: almost 6% of screened in participants outright declined to complete a survey about their innovative activities. These refusals are also troubling because it is plausible that refusals are correlated with the type or quality of innovation. Qualitatively, a few participants refused to describe their innovation in detail or commented that they were suspicious about the true purpose of the survey. Suspicion could be less of an issue for a federal statistical survey that is more readily viewed as credible, but distrust of providing this kind of information online may contribute to the lower innovation rates obtained from Web surveys than from telephone or in-person surveys (de Jong, 2016; Franke, Schirg, and Reinsberger 2016).

Conclusion

Individuals have recently been recognized as an important source of innovation. Using the OECD definition of innovation, we determined that most respondents could usually differentiate innovation from non-innovation. Respondents could also discuss their own innovations. The incidence rate of individual innovation on MTurk is high enough to recruit moderately sized samples of innovators. These innovators have diverse experiences with innovations and have characteristics that are broadly similar to those observed in other samples, making it a potentially useful tool for pilot-testing survey instruments about innovation. Using a crowdsourcing tool for respondent recruitment and survey administration allowed us to conduct this survey. This project would have been prohibitively expensive without using MTurk as it would require extensive screening. One challenge that would have to be overcome in future studies is the potential reluctance of participants to share details of their innovations. Another challenge is how to extrapolate findings from the MTurk sample to a specific population of interest.

The study presented here also suggests that a more rigorous screening procedure can impact the incidence rates of individual innovation, resulting in far fewer people screening into the study than would likely be removed in subsequent data processing. This represents a proof of concept, but there are many design choices that could improve these screening criteria.

References

Anderson CA, Allen JJ, Plante C, Quigley-McBride A, Lovett A, Rokkum JN. 2019. The MTurkification of Social and Personality Psychology. Personality and Social Psychology Bulletin 45:842–850. doi: 0146167218798821.

Casey LS, Chandler J, Levine AS, Proctor A, Strolovitch DZ. 2017. Intertemporal Differences among MTurk Workers: Time-Based Sample Variations and Implications for Online Data Collection. SAGE Open 7.

Chandler J, Shapiro D. 2016. Conducting Clinical Research Using Crowdsourced Convenience Samples. Annual Review of Clinical Psychology 12:53–81.

de Jong JP. 2016. Surveying Innovation in Samples of Individual End Consumers. European Journal of Innovation Management 19:406–23. doi:10.1108/ EJIM-09-2015-0093.

de Jong JP, von Hippel E, Gault F, Kuusisto J, Raasch C. 2015. Market Failure in the Diffusion of Consumer-Developed Innovations: Patterns in Finland. Research Policy 44:1856–65.

Edgar J, Murphy J, Keating M. 2016. Comparing Traditional and Crowdsourcing Methods for Pretesting Survey Questions. SAGE Open 6.

Franke, N., Schirg, F., Reinsberger, K., The frequency of end-user innovation: A re-estimation of extant ﬁndings. Research Policy (2016), http://dx.doi.org/10.1016/j.respol.2016.04.012.

Miranda J, Zolas N. 2018. Measuring the Impact of Household Innovation Using Administrative Data. NBER Working Papers 25259. Cambridge, MA: National Bureau of Economic Research.

Ogawa S, Pongtanalert K. 2011. Visualizing Invisible Innovation Content: Evidence from Global Consumer Innovation Surveys. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1876186. Accessed 9 July 2019.

Peer E, Vosgerau J, Acquisti A. 2014. Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk. Behavior Research Methods 46:1023–31.

Stewart N, Chandler J, Paolacci G. 2015. Crowdsourcing Samples in Cognitive Science. Trends in Cognitive Sciences 21:736–48.

von Hippel E, de Jong JP, Flowers S. 2012. Comparing Business and Household Sector Innovation in Consumer Products: Findings from a Representative Study in the United Kingdom. Management Science 58:1669–81.

von Hippel E, Ogawa S, de Jong JP. 2011. The Age of the Consumer-Innovator. MIT Sloan Management Review 53:27–35.

Notes

1Organisation for Economic Co-operation and Development (OECD) and Statistical Office of the European Communities (Eurostat). 2018. Oslo Manual: Guidelines for Collecting, Reporting and Using Data on Innovation. 4th ed., p. 20. Paris.

2Organisation for Economic Co-operation and Development (OECD) and Statistical Office of the European Communities (Eurostat). 2018. Oslo Manual: Guidelines for Collecting, Reporting and Using Data on Innovation. 4th ed., p. 20. Paris.

3von Hippel E. “Democratizing Innovation: The evolving phenomenon of user innovation.” An excerpt from the book Democratizing Innovation published by MIT Press. Available at https://www.oecd.org/sti/inno/37450155.pdf.

Suggested Citation

Chandler J, Kindlon A, Morrison R; National Center for Science and Engineering Statistics (NCSES). 2022. Using Mechanical Turk in Study of Individual Innovation. Working Paper NCSES 22-201. Alexandria, VA: National Science Foundation. Available at https://www.nsf.gov/statistics/2022/ncses22201/.

Contact

Report Authors

Jesse Chandler
Survey Researcher
Mathematica, under contract to NCSES

Audrey Kindlon
Survey Statistician
Research and Development Statistics Program, NCSES
Tel: (703) 292-2332
E-mail: akindlon@nsf.gov

Rebecca Morrison
Survey Statistician
Statistics and Methods Program, NCSES
Tel: (703) 292-7794
E-mail: rlmorris@nsf.gov

NCSES

National Center for Science and Engineering Statistics
Directorate for Social, Behavioral and Economic Sciences
National Science Foundation
2415 Eisenhower Avenue, Suite W14200
Alexandria, VA 22314
Tel: (703) 292-8780
FIRS: (800) 877-8339
TDD: (800) 281-8749
E-mail: ncsesweb@nsf.gov

Appendix A

Appendix A. Phase 1 Survey Instrument

Section A – Introduction

A0 – Embedded Data Field

[Save variable workerId from URL, case sensitive]

A1 - Introduction

[programmer include NSF logo left justified, NCSES logo right justified]

This survey is conducted by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF). This survey is being collected by NCSES under OMB No. 3145-0174. This survey will take approximately 20 minutes to complete. Your participation is voluntary and you have the right to stop at any time.

Please take your time as you answer these questions. The information you provide will contribute to valuable research at NCSES, one of the principal Federal statistical agencies.

This survey is being administered by Mathematica Policy Research, Inc. and resides on a server outside of the NCSES domain. NCSES cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding, you give your consent to participate in this survey.

[page break]

A1 - Instructions

For the purpose of this survey, we would like you to use the following definition for innovation:

Using this definition, please classify the following examples as innovation or not innovation. You will be able to review this definition at any time by clicking the word “innovation or “innovative”.

Section B - Vignettes

[Select six product innovations without replacement, for each, randomly assign the respondent to see either innovative or not innovative vignettes and randomly assign the person described to have a male or female name]

[Select one process innovation and randomly assign the respondent to see either innovative or not innovative vignettes and randomly assign the person described to have a male or female name]

[Randomly assign the order in which the seven selected vignettes will appear]

B.[vignette_number].1 – Is Activity Innovative

[insert vignette here]

Do you consider the activity described an innovation?

1 – Yes

2 – No

B.[vignette_number].2 – Confidence

[insert vignette here]

How confident are you that your classification of the activity as [if Q1 = 1 “as innovative”; if Q1 = 2, “as not innovative”] is correct?

1 – Not at all confident

2 – Somewhat confident

3 – Very confident

B.[vignette_number].3 – Rationale

Please tell us why you chose to classify the example as [if B.[vignette_number].2 = 1 “as innovative”; if B.[vignette_number]2 = 2, “as not innovative”]

[Open Answer]

Section C – Demographics

C.1 Sex

Are you…

1 – Male

2 – Female

C.2 Age

What is your age?

Age ___

C.3 Residence

Are you living in the United States, Puerto Rico, or another U.S. territory, or are you living in another country?

1 – United States, Puerto Rico, or another U.S. territory

2 – Another country

C.4 Thank you

Thank you for your time.

Appendix B

Appendix B. Phase 1 Vignettes

Product Innovations

Process Innovations

Number	Individual innovation	Not individual innovation	Reason not innovative
19	Sarah (Christopher) is a small business owner and to maintain copies of her (his) receipts for her (his) business she (he) developed a program that takes her (his) electronic credit card statement and transfers each transaction into an Excel workbook where she (he) can enter the additional information about each transaction that she (he) will need for her (his) business’s tax return.	Sarah (Christopher) is a small business owner and to maintain copies of her (his) receipts for her (his) business she (he) uses her (his) smartphone to take a photograph of each receipt. At the end of the year she (he) provides all the photographs to her (his) accountant for her (his) business’s tax return.	Did not create something new
20	Karen (Neil) is a high school chemistry teacher and one weekend, while working on some experiments for her (his) class, she (he) develops a new way to produce plastic without using petroleum.	Karen (Neil) is a high school chemistry teacher and one weekend, while working on some experiments for her (his) class and consulting the internet she (he) figures out how to produce plastic at home.	Did not create something new
21	Betty (Samson) used three pots to create a multiple level pot that allows waste heat from the bottom pot to be used to cook food in the upper level pots.	Betty (Samson) used two pots to create a double boiler because she (he) did not have one at home.	Homemade version of something commercially available
22	Sandra (Julian) sells cupcakes at a local farmer’s market on weekends. To keep up with growing demand, she (he) started baking her (his) cupcakes in cupcake liners placed in mason jar lids on baking sheets. This way, she (he) does not need to wait for the tray to cool before removing the cupcakes out and reusing it.	Sandra (Julian) sells cupcakes at a local farmer’s market on weekends. To keep up with growing demand, she (he) purchased additional muffin tins. This way, she (he) can bake a second batch of cupcakes while waiting for the first batch to cool.	Did not create something new

Appendix C

Appendix C. Phase 2 Survey Instrument

A1 - Introduction

This survey is conducted by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF). This survey is being collected by NCSES under OMB No. 3145-0174. This survey will take approximately three minutes to complete. Your participation is voluntary. The information you provide will contribute to valuable research at NCSES, one of the principal Federal statistical agencies.

[page break]

B1 – Sex

Are you

1 – Male

2 – Female

[page break]

B2 – Age

What is your age?

_______ years old

[hard check age]

[if 18, go to end of survey]

[page break]

B3 – Residence

Where do you live?

1 – A US state (including the District of Columbia)

2 – Puerto Rico or another US territory

3 – Another country

[hard check residence]

[if 3, go to end of survey]

B4 – State [display if B3 = 1]

What is your state of legal residence?

[Drop down box, use FIPS codes]

1 AL

2 AK

5 AR

4 AZ

6 CA

8 CO

9 CT

11 DC

10 DE

12 FL

13 GA

15 HI

16 ID

17 IL

18 IN

19 IA

20 KS

21 KY

22 LA

23 ME

24 MD

25 MA

26 MI

27 MN

28 MS

29 MO

30 MT

31 NE

32 NV

33 NH

34 NJ

35 NM

36 NY

37 NC

38 ND

39 OH

40 OK

41 OR

42 PA

44 RI

45 SC

46 SD

47 TN

48 TX

49 UT

50 VT

51 VA

53 WA

54 WV

55 WI

56 WY

B5 – Territory [display if B3 = 2]

What is your territory of legal residence?

[Drop down box, uses FIPS codes]

60 AS

66 GU

69 MP

72 PR

78 VI

99 Other (Specify)

[page break]

B6 – Degree

What is the highest degree or level of school you have completed?

1 – Less than high school/secondary school graduate

2 – Regular high school diploma

3 – GED or alternative credential

4 – Some college credit, but less than 1 year of college credit

5 – 1 or more years of college credit, no degree

6 – Associate’s degree (for example: AA, AS)

7 – Bachelor’s degree (for example: BA, BS)

8 – Master’s degree (for example: MA, MS, MEng, MEd, MSW, MBA)

9 – Professional degree beyond a bachelor’s degree (for example: MD, DDS, DVM, LLB, JD)

10 – Doctorate degree (for example: PhD, EdD)

99 – Not applicable/Unknown

[page break]

B7 – Ethnicity

Are you of Hispanic, Latino, or Spanish origin?

0 – No, not of Hispanic, Latino, or Spanish origin

1 – Yes, Mexican, Mexican American or Chicano

2 – Yes, Puerto Rican

3 – Yes, Cuban

4 – Yes, another Hispanic, Latino, or Spanish origin

[page break]

B8 – Race

What is your race?

Select all that apply.

1 – White

2 – Black or African American

3 – American Indian or Alaska Native

4 – Asian

5 – Native Hawaiian or Other Pacific Islander

[page break]

F1 – Innovations

The next section of the survey will ask you questions about personal innovations. Personal innovations are innovations that you have developed on your own time. An innovation is a new or improved product (good or service) or process that:

Is not made for work (but a person can use their professional experience, skills or knowledge to create something on their own time).
Is used by the developer(s) or has been made available to other potential users.
Is new or different in some way, and not a home-built version of an existing product or process currently on the market. However, innovations may include modifications of existing products or processes to create something functionally novel.

Before we begin, please classify the following examples as personal innovation or not personal innovation to make sure that you understand this definition.

F1a – Innovation Test 1

Larry is a farm tool designer. During his lunch hour, he designed and built a device that allows him to pull weeds in his garden without bending down.

1 – Personal Innovation

0 – Not Personal Innovation

[if F1a = 1 display] Correct. Larry’s new weed puller is innovative if he designed and built it outside of his duties for work.

[if F1a = 0 display] Incorrect. Larry’s new weed puller is a novel product that he designed and built on his own time and that he uses. If he developed it as a part of his work duties, we would not consider it personal innovation.

[page break]

F1b – Innovation Test 2

The online sign-up for a local event was first come, first served and became oversubscribed quickly. Patricia, a stay-at-home parent, created a computer program from scratch that, once running, reloaded the sign-up page until it went live at midnight and then quickly filled in all the information, enabling Patricia to register without staying up late. The program is integrated with her calendar, so it will not book tickets on a day that she is busy.

1 – Personal Innovation

0 – Not Personal Innovation

[if F1b = 1 display] Correct. Patricia’s new computer program is innovative because it is new or different in some way from others that she knows exist. It would also be innovative if it was a new application of an existing product. If Patricia knew a program like this existed, and she tried to exactly copy both its purpose and how it achieved this purpose, we would not consider it personal innovation.

[if F1b = 0 display] Incorrect. Patricia’s new computer program is a novel product that she built on her own time and that she uses. If Patricia knew a program like this existed, and she tried to copy both its purpose and how it achieved this purpose, we would not consider it personal innovation.

[page break]

F1c – Innovation Test 3

Frustrated with the inability to program commercial music streaming services, Michael adapted a service so he would wake up to different genres of music according to the weather. For example, when it was raining, the algorithm would play up-tempo and energetic music. Though Michael adapted a music streaming service to his needs, he found another product he liked better, and never used his own product.

1 – Personal Innovation

0 – Not Personal Innovation

if F1a = 1 display] Incorrect. Michael’s development of a music streaming service program is a novel modification of a product that he made in his own time. However, for the purpose of this survey, we are only interested in innovations that the developer uses or makes available to others. Since he never used it, it is not innovative.

if F1a = 0 display] Correct. Michael’s development of a music streaming service is a novel modification of a product that he made in his own time. However, for the purpose of this survey we are only interested in innovations that the developer uses or makes available to others. Since he never used it, it is not innovative.

[page break]

C1 – Create Innovation?

We would like you to think about whether you have created any personal innovations of your own. During the three years 2016 to 2018, did you create a new or improved good or service?

1 – Yes

0 – No [set “Eligible” to 0, go to end of survey]

[page break]

C2 – Create Innovation for Work?

Was this new or improved good or service created for your work or on your own time?

3 – I created new or improved goods and services both for work and on my own time

1 – I created new or improved goods and services for work only [ set “Eligible” to 0, go to end of survey]

0 – I created new or improved goods and services on my own time only

[page break]

C4 – Improvement of Existing Product?

Was this new or improved good or service a home-built version of an existing product or process currently on the market?

1 – Yes [ set “Eligible” to 0, go to end of survey]

0 – No

[page break]

C5 – Modify Product?

Did this new or improved good or service modify products or processes to create something functionally novel?

1 – Yes

0 – No

[page break]

C6 – Completely New Product?

Was this new or improved good or service a completely new good or service not available on the market?

1 – Yes, it was a new or improved good or service

0 – No, it was not a new or improved good or service

[if C5 = 0 and C6 = 0 set “Eligible” to 0, go to end of survey]

[page break]

C7 – Who Uses Innovation?

Do you use this new good or service or has it been made available to others?

1 – I use it and it has been made available to others

2 – I use it but it has not been made available to others

3 – I don’t use it but it has been made available to others

4 – I do not use it and it is not available for others to use [ set “Eligible” to 0, go to end of survey]

D1 – Innovation Invitation

Based on your responses, you are eligible to complete an additional survey about personal innovation. This survey will take approximately 20 minutes to complete. If you complete this survey, you will be awarded a bonus payment of $3.00 in addition to your payment for completing the first survey.

Are you interested in completing this survey?

1 – Yes, I am interested in completing an additional survey for a bonus payment of $3.00 [go to E1]

2 – No, but I would be willing to be invited to complete this survey at a later date [go to D2]

0 – No, I am not interested in completing an additional survey [go to J1]

[page break]

D2 - Closing Comments

Is there anything else you want to tell us before you complete the survey?

____________

[Go to J1]

E1 – Innovation Introduction

This survey is conducted by the National Center for Science and Engineering Statistics (NCSES) within the National Science Foundation (NSF). This survey is being collected by NCSES under OMB No. 3145-0174. This survey will take approximately 15 minutes to complete. Your participation is voluntary and you have the right to stop at any time.

Please take your time as you answer these questions. The information you provide will contribute to valuable research at NCSES, one of the principal Federal statistical agencies.

[page break]

F2a – Software 1

[display innovation definition when “create” is clicked on]

Earlier, you told us that you have created personal innovations of your own. We would like to learn more about the personal innovations that you have created. You can review the definition of personal innovation by clicking on the word “create” below.

Within the past three years, did you ever use your own time to create your own computer software program including mobile or web apps?

1 – Yes

0 – No

[page break]

F2b – Software 2 [display if F2a = 1]

Did you create multiple computer software products including mobile or web apps during your own time in the last three years?

1 – Yes

0 – No

[page break]

F2c – Software 3 [display if F2b = 1]

How many computer software products including mobile or web apps have you created on your own time during the last three years?

Number of software products ___

[page break]

F3a – Household 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create your own household fixture or furnishing, such as kitchen- and cookware, cleaning devices, lighting, furniture, and more?

1 – Yes

0 – No

[page break]

F3b – Household 2 [display if F3a = 1]

Did you create multiple household fixtures or furnishings during your own time in the last three years?

1 – Yes

0 – No

[page break]

F3c – Household 3 [display if F3b = 1]

How many household fixtures or furnishings have you created on your own time during the last three years?

Number of household fixture or furnishings ___

[page break]

F4a – Transport 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create your own transport or vehicle-related product or part, such as for cars, bicycles, scooters or anything related?

1 – Yes

0 – No

[page break]

F4b – Transport 2 [display if F4a = 1]

Did you create multiple transport or vehicle-related products or parts during your own time in the last three years?

1 – Yes

0 – No

[page break]

F4c – Transport 3 [display if F4b = 1]

How many transport or vehicle-related products or parts have you created on your own time during the last three years?

Number of transport or vehicle-related product or parts ___

[page break]

F5a – Tools 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create your own tools or equipment, such as utensils, molds, gardening tools, mechanical or electrical devices, and so on?

1 – Yes

0 – No

[page break]

F5b – Tools 2 [display if F5a = 1]

Did you create multiple tools or pieces of equipment during your own time in the last three years?

1 – Yes

0 – No

[page break]

F5c – Tools 3 [display if F5b = 1]

How many tools or pieces of equipment have you created on your own time during the last three years?

Number of tools or equipment ___

[page break]

F6a – Entertainment 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create your own sports-, hobby- or entertainment product, such as sports devices or games?

1 – Yes

0 – No

[page break]

F6b – Entertainment 2 [display if F6a = 1]

Did you create multiple sports-, hobby- or entertainment products during your own time in the last three years?

1 – Yes

0 – No

[page break]

F6c – Entertainment 3 [display if F6b = 1]

How many sports-, hobby- or entertainment products have you created on your own time during the last three years?

Number of sports-, hobby- or entertainment products ___

[page break]

F7a – Education 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create your own children- or education-related product, such as toys and tutorials?

1 – Yes

0 – No

[page break]

F7b – Education 2 [display if F7a = 1]

Did you create multiple children- or education-related products during your own time in the last three years?

1 – Yes

0 – No

[page break]

F7c – Education 3 [display if F7b = 1]

How many children- or education-related products have you created on your own time during the last three years?

Number of children- or education-related products ___

[page break]

F8a – Care 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create your own help-, care- or medical-related product?

1 – Yes

0 – No

[page break]

F8b – Care 2 [display if F8a = 1]

Did you create multiple help-, care- or medical-related products during your own time in the last three years?

1 – Yes

0 – No

[page break]

F8c – Care 3 [display if F8b = 1]

How many help-, care- or medical-related products have you created on your own time during the last three years?

Number of help-, care- or medical-related products ___

[page break]

F9a – Other Product 1

[display innovation definition when “create” is clicked on]

Within the past three years, did you ever use your own time to create any other types of products?

1 – Yes

0 – No

[page break]

F9b – Other Product 2 [display if F9a = 1]

Did you create multiple other types of products during your own time in the last three years?

1 – Yes

0 – No

[page break]

F9c – Other Product 3 [display if F9b = 1]

How many other types of products have you created on your own time during the last three years?

Number of other types of products ___

[page break]

Sum yes responses in F2a, F3a, F4a, F5a, F6a, F7a, F8a and F9a as InnovationCount. If InnovationCount = 0 go to J1, if 1 fill to F10 and go to G1, if [1 go to F10]

F10 – Multiple Innovations

You just identified a number of products you have made in more than one area. Which one do you consider most significant?

2 – [display if F2a = 1] Computer software

3 – [display if F3a = 1] Household fixture or furnishing

4 – [display if F4a = 1] Transport or vehicle-related product or part

5 – [display if F5a = 1] Tool or piece of equipment

6 – [display if F6a = 1] Sports-, hobby- or entertainment product

7 – [display if F7a = 1] Children- or education-related product

8 – [display if F8a = 1] Help-, care- or medical-related product

9 – [display if F9a = 1] Other types of product

G1 – Your Primary Creation [ display if ] 0 of F2a, F3a, F4a, F5a, F6a, F7a, F8a and F9a is selected]

The next questions ask about a [Fill F10] product. If you created more than one [fill F10] product, think about the one that is most important to you.

[page break]

G2 – Describe Innovation

What kind of [Fill F10] did you create?

__________

[page break]

G3 – Innovation Novelty

What was new about this [Fill F10]?

__________

[page break]

G4 –Substitutability

At the time you developed the [Fill F10], could you have bought ready-made similar products on the market?

1 – Yes

0 – No

99 – Don’t know

[page break]

G5 – Still Use

Do you still use the [Fill F10]?

1 – Yes, I still use it

2 – No, I used to use it but don’t anymore

0 – No, I never used it

[page break]

G6 – Motivation - Personal Need

Did you develop this [Fill F10] because you personally needed it?

1 – Yes

0 – No

[page break]

G7 – Motivation - Make Money

Did you develop this [Fill F10] because you want to sell it or make money?

1 – Yes

0 – No

[page break]

G8 – Motivation - Develop Skills

Did you develop this [Fill 10] because you wanted to learn or to develop your skills?

1 – Yes

0 – No

[page break]

G9 – Motivation - Helping People

Did you develop this [Fill 10] because you were helping other people?

1 – Yes

0 – No

[page break]

G10 – Motivation - Fun

Did you develop this [Fill F10] for the fun of doing it?

1 – Yes

0 – No

[page break]

G11a – Collaboration

Did you work with other people to develop this [Fill F10]?

1 – Yes

0 – No

[page break]

G11b – Number of Collaborators [display if G11a = 1]

Approximately how many others contributed to developing this [insert innovation]?

Number of people: __________

[page break]

G12 – Investment in Innovation

Did you spend any money developing this [insert innovation]?

1 – Yes

0 – No

[page break]

H1 – Protecting IP

Did you use any of the following methods to protect this [Fill F10]?

a. Utility patents(patents for inventions)	1 – Yes	0 – No
b. Design patents (patents for appearance)	1 – Yes	0 – No
c. Trademarks	1 – Yes	0 – No
d. Copyrights	1 – Yes	0 – No
e. Trade secrets	1 – Yes	0 – No
f. Nondisclosure agreements	1 – Yes	0 – No

[page break]

H3 – Innovation Demonstration

Did you do anything to inform other people or businesses about your [Fill F10]? (For example: showing it off, communicating about it or, posting its design to the web)

1 – Yes

0 – No

[page break]

H4 – Innovation Adoption

To the best of your knowledge, have any other people adopted your [Fill F10] for personal use?

1 – Yes

0 – No

[page break]

H5 – Contact Others [display if H4 = 0]

Do you intend to contact other people who may adopt your [Fill F10] for personal use?

1 – Yes

0 – No

[page break]

I1 – Business Owner

Do you, alone or with others, currently own a business you help manage, or are you self-employed?

1 – Yes

0 – No

[page break]

I2 – Sell Through Business [display if I1 = 1]

Did you commercialize your [Fill F10] via your business? Or do you intend to do this?

1 – Yes, I commercialized it

2 – I intend to do so

0 – No

[page break]

I3 – Starting New Business [display if I2 = 0]

Are you currently, alone or with others, trying to start a new business?

1 – Yes

0 – No

[page break]

I4 – Intent to Commercialize [display if I3 = 1]

Do you intend to commercialize your [fill F10] with this new business?

1 – Yes

0 – No

[page break]

I5 – Innovation Adoption

Commercial businesses may be interested in your [fill F10]. Did any commercial business buy or license your [fill F10] with intent to sell?

1 – Yes

0 – No

[page break]

I6 – Intend to Sell [display if I5 = 0]

Do you intend to contact commercial businesses to adopt your [fill F10] for general sale?

1 – Yes

0 – No

[page break]

E2 – Urban or Rural

Which of the following best describes the area you live in?

1 – Urban – a city

2 – Suburban – a smaller community adjacent to or within commuting distance of a city

3 – Rural – the country or an area located outside towns and cities

[page break]

E3 – People in Household

How many people lived or stayed in your household on March 1st, 2019?

INCLUDE everyone who is living or staying in your household for more than 2 months.
INCLUDE anyone else staying in your household who does not have another place to stay, even if they are there for 2 months or less.
DO NOT INCLUDE anyone who is living somewhere else for more than 2 months, such as a college student living away or someone in the Armed Forces on deployment.

Number of people _________

[page break]

E4a – Disability 1

The next questions ask about difficulties you may have doing certain activities because of a HEALTH PROBLEM.

Do you have difficulty seeing, even if wearing glasses?

0 – No

1 – Yes

E4b – Disability 2

Do you have difficulty hearing, even if using a hearing aid?

0 – No

1 – Yes

E4c – Disability 3

Do you have difficulty walking or climbing steps?

0 – No

1 – Yes

E4d – Disability 4

Do you have difficulty remembering or concentrating?

0 – No

1 – Yes

E4e – Disability 5

Do you have difficulty (with self-care such as) washing all over or dressing?

0 – No

1 – Yes

E4f – Disability 6

Using your usual (customary) language, do you have difficulty communicating, for example understanding or being understood?

0 – No

1 – Yes

[page break]

E5a – Employment

Were you working for pay or profit during the past week?

Working includes being self-employed and not getting paid that week, on a postdoctoral appointment, traveling while employed, or on any type of paid or unpaid leave, including vacation.

1 – Yes

0 – No

[page break]

E5b – Looking for Employment [display if E5a = 0]

Did you look for work during the four weeks preceding today?

1 – Yes

0 – No

[page break]

E6 – Job Category

What is the job category that best describes the last job you held prior to the week of [fill survey posting date, TBD]?

To see a definition of each job category, click on the text.

1 – Management Occupations

Hover Text: [Top Executives; Advertising, Marketing, Promotions, Public Relations, and Sales Managers; Operations Specialties Managers; Other Management Occupations]

2 – Business and Financial Operations Occupations

Hover Text: [Business Operations Specialists; Financial Specialists]

3 – Computer and Mathematical Occupations

Hover Text: [Computer Occupations; Mathematical Science Occupations]

4 – Architecture and Engineering Occupations

Hover Text: [Architects, Surveyors, and Cartographers; Engineers; Drafters, Engineering Technicians; and Mapping Technicians]

5 – Life, Physical, and Social Science Occupations

Hover Text: [Life Scientists; Physical Scientists; Social Scientists and Related Workers; Life, Physical, and Social Science Technicians; Occupational Health and Safety Specialists and Technicians]

6 – Community and Social Service Occupations

Hover Text: Hover Text: [Counselors, Social Workers, and Other Community and Social Service Specialists; Religious Workers]

7 – Legal Occupations

Hover Text: [Lawyers, Judges, and Related Workers; Legal Support Workers]

8 – Educational Instruction and Library Occupations

Hover Text: [Postsecondary Teachers; Preschool, Elementary, Middle, Secondary, and Special Education Teachers; Other Teachers and Instructors; Librarians, Curators, and Archivists; Other Educational Instruction and Library Occupations]

9 – Art, Design, Entertainment, Sports, and Media Occupations

Hover Text: [Art and Design Workers; Entertainers and Performers, Sports and Related Workers; Media and Communication Workers; Media and Communication Equipment Workers]

10 – Healthcare Practitioners and Technical Occupations

Hover Text: [Healthcare Diagnosing or Treatment Practitioners; Health Technologists and Technicians; Other Healthcare Practitioners and Technical Occupations]

11 – Healthcare Support Occupations

Hover Text: [Home Health and Personal Care Aides; and Nursing Assistants, Orderlies, and Psychiatric Aides; Occupational Therapy and Physical Therapist Assistants and Aides; Other Healthcare Support Occupations]

12 – Protective Service Occupations

Hover Text: [Supervisors of Protective Service Workers; Firefighting and Prevention Workers; Law Enforcement Workers; Other Protective Service Workers]

13 – Food Preparation and Serving Related Occupations

Hover Text: [Supervisors of Food Preparation and Serving Workers; Cooks and Food Preparation Workers; Food and Beverage Serving Workers; Other Food Preparation and Serving Related Workers]

14 – Building and Grounds Cleaning and Maintenance Occupations

Hover Text: [Supervisors of Building and Grounds Cleaning and Maintenance Workers; Building Cleaning and Pest Control Workers; Grounds Maintenance Workers]

15 – Personal Care and Service Occupations

Hover Text: [Supervisors of Personal Care and Service Workers; Animal Care and Service Workers; Entertainment Attendants and Related Workers; Funeral Service Workers; Personal Appearance Workers; Baggage Porters, Bellhops, and Concierges; Tour and Travel Guides; Other Personal Care and Service Workers]

16 – Sales and Related Occupations

Hover Text: [Supervisors of Sales Workers; Retail Sales Workers; Sales Representatives, Services; Sales Representatives, Wholesale and Manufacturing; Other Sales and Related Workers]

17 – Office and Administrative Support Occupations

Hover Text: [Supervisors and Office and Administrative Support Workers; Communications Equipment Operators; Financial Clerks; Information and Record Clerks; Material Recording, Scheduling, Dispatching, and Distributing Workers; Secretaries and Administrative Assistants; Other Office and Administrative Support Workers]

18 – Farming, Fishing, and Forestry Occupations

Hover Text: [Supervisors of Farming, Fishing, and Forestry Workers; Agricultural Workers; Fishing and Hunting Workers; Forest, Conservation, and Logging Workers]

19 – Construction and Extraction Occupations

Hover Text: [Supervisors of Construction and Extraction Workers; Construction Trades Workers; Helpers, Construction Trades; Other Construction and Related Workers; Extraction Workers]

20 – Installation, Maintenance, and Repair Occupations

Hover Text: [Supervisors of Installation, Maintenance, and Repair Workers; Electrical and Electronic Equipment Mechanics, Installers, and Repairers; Vehicle and Mobile Equipment Mechanics, Installers, and Repairers; Other Installation, Maintenance, and Repair Occupations]

21 – Production Occupations

Hover Text: [Supervisors of Production Workers; Assemblers and Fabricators; Food Processing Workers; Metal Workers and Plastic Workers; Printing Workers; Textile, Apparel, and Furnishings Workers; Woodworkers; Plant and System Operators; Other Production Occupations]

22 – Transportation and Material Moving Occupation

Hover Text: [Supervisors of Transportation and Material Moving Workers; Air Transportation Workers; Motor Vehicle Operators; Rail Transportation Workers, Water Transportation Workers; Other Transportation Workers; Material Moving Workers]

23 – Military Specific Occupations

Hover Text: [Military Officer Special and Tactical Operations Leaders; First-Line Enlisted Military Supervisors; Military Enlisted Tactical Operations and Air/Weapons Specialists and Crew Members]

24 – Other Occupations Not Specified

[page break]

E7 – Household Income

Which of the following categories best describes your total household income from all sources, before taxes and deductions, in 2017?

1 – Less than $25,000

2 – $25,000 to $49,999

3 – $50,000 to $74,999

4 – $75,000 to $99,999

5 – $100,000 to $124,999

6 – $125,000 to $144,999

7 – $150,000 or higher

99 – Decline to answer

[page break]

J1 - Closing Comments

Is there anything else you want to tell us before you complete the survey?

__________

[page break]

J2 - Complete

Thank you for your time. Your confirmation number is [Fill unique confirmation number]