Skip To Content

Table Of Contents

Click for DHHS Home Page
Click for the SAMHSA Home Page
Click for the OAS Drug Abuse Statistics Home Page
Click for What's New
Click for Recent Reports and HighlightsClick for Information by Topic Click for OAS Data Systems and more Pubs Click for Data on Specific Drugs of Use Click for Short Reports and Facts Click for Frequently Asked Questions Click for Publications Click to send OAS Comments, Questions and Requests Click for OAS Home Page Click for Substance Abuse and Mental Health Services Administration Home Page Click to Search Our Site

Computer Assisted Interviewing for SAMHSA's National Household Survey on Drug Abuse

10. Willingness to Respond

In Chapter 5, it was noted that the field experiment response rates were low compared with those of the comparison group from Quarter 4 of the 1997 NHSDA. In this chapter, we examine these response rate differences in more depth, and we seek reasons for the lower response rates experienced during the 1997 field experiment.

10.1 Response Rates

The field experiment's response rates were considerably lower than the Quarter 4 NHSDA rates. Exhibit 5.4.1 shows that the field experiment's screening response rate was 85.4% compared with 93.6% for the comparison group. Exhibit 5.4.2 shows that the field experiment's person response rate of 62.7% was 12.9% lower than the overall person-level response rate attained in the comparison group (75.6%). Results of the field experiment for the electronic survey instruments should be evaluated by comparing them with those obtained from Quarter 4 of 1997 NHSDA using the standard paper-and-pencil interviewing (PAPI) instrumentation. However, the two surveys used separate field supervisor (FS) and field interviewer (FI) staffs, confounding the comparisons of electronic versus PAPI instrumentation differences with those due to the differences in the two interviewing staffs. The survey response rates also may have been affected by the attitudes of the interviewers and their supervisors toward the 1997 field experiment survey, as well as the attitudes of the respondents themselves toward the electronic instruments.

10.1.1 Dwelling Unit Screening

We examined the screening response rates by region, metropolitan statistical area (MSA) size, and certainty/noncertainty area. These details are shown in Exhibit 10.1.1. The dwelling unit (DU) ineligibility rates for the field experiment were quite similar to those for the 1997 NHSDA Quarter 4 comparison group. The same was largely the case for the ineligibility rates across the subgroups examined. A few differences in the subgroup ineligibility rates were observed, with somewhat higher vacant DU rates during the field experiment in the Northeast region and in large MSAs. Overall, however, the ineligibility rates for the field experiment and the comparison group sample were remarkably similar.

There were major differences between the two surveys in the percentage of eligible DUs that were successfully screened. The field experiment screening rate of 85.4% was 8.2 percentage points below the 93.6% rate for the comparison group. Exhibit 10.1.2 shows that nearly all of this large shortfall was associated with much higher occurrences of field experiment screening refusals (i.e., units in which the interviewers were denied access) and the other nonresponse category. Each of these three nonresponse rates were at least twice as high for the field experiment as they were for the comparison group. The two rates of nonresponse for not at homes were nearly identical. Units not screened because of Newton screener problems, a special field experiment nonresponse code, accounted for less than 1% of the eligible DUs.

Examination of the detailed data in Exhibit 10.1.1 reveals some contrasting patterns in the DU nonresponse subcategories. The high refusal rates in the field experiment occurred across all of the subgroups. The nonscreens due to denied access, however, occurred predominantly in the West region (8.2%), large MSAs (5.5%), and certainty primary sampling units (PSUs) (5.6%). The field experiment's denied access rates for those subgroups were many times larger than the rates for all other subgroups. Interestingly, the denied access rates for the 1997 NHSDA Quarter 4 comparison study were very low for all of these subgroups (generally 1% or less). Denied access is a highly variable reason for nonresponse and includes gated communities and apartment buildings that do not allow interviewers to directly visit the sample house or apartment. When the survey team has been working in an area for a considerable amount of time, they develop skills and contacts that allow them to get past the gatekeepers. Given that the interviewing staff who were working in the field experiment were new to the NHSDA and worked for only a short period of time, it is not surprising that this rate was higher than it was in the comparison group.

Still another pattern of subgroup nonresponse was present in the "other nonresponse" category. The field experiment's other nonresponse rates were considerably higher than the comparison sample rates, both overall and for subgroups. The majority of the excess other nonresponse in the field experiment was due to the group quarters. The screener application did not include the group quarters screening algorithm, and these types of DUs were coded as "other" when assigning an outcome code during the screening. Overall, about 1.4% of the DUs were group quarters. Removing these DUs from the other nonresponse category results in similar rates for the field experiment and the comparison group. Reflective of an uneven distribution of group quarters, there was variability across areas: the Midwest (5.1%), the South (3.3%), non-MSA areas (7.5%), and noncertainty PSUs (4.2%) in this other nonresponse.

10.1.2 Person-Level Nonresponse

The field experiment's overall person response rate was 62.7%, which was 12.9 percentage points lower than the 75.6% response-obtained rate from the 1997 NHSDA Quarter 4 survey. The detailed person nonresponse rate information for all sample persons, from the field experiment and the comparison survey is shown in Exhibit 10.1.3. There was no Spanish version of the electronic questionnaire for the field experiment. Thus, for comparability of data in this report, cases that were completed in Spanish in the NHSDA comparison group were coded as language barrier nonresponses in these tabulations. We must keep this in mind when considering the field experiment's person response rates.

The lower person response rate obtained from the field experiment was primarily due to the field experiment's 19.8% person refusal rate in contrast to the 8.6% person refusal rate for the comparison survey. Although the refusal rate accounted for the largest difference in the person response rates, the field experiment also encountered a somewhat higher nonresponse for physical/mental disabled and for other nonresponse reasons (see Exhibit 10.1.4). These were partially offset by somewhat lower field experiment nonresponse rates for the no-one-home and language-barrier categories. These differences were considerably less dramatic than the refusal rate difference described above.

The response rates for the field experiment and comparison survey exhibited many similar patterns of nonresponse across various subpopulations examined. In both surveys, the following statements can be made concerning the pattern of person-level response rates:

  1. Response was negatively correlated with respondent age.

  2. Response was lower for Hispanics than for blacks or whites and others.

  3. Female response rate was slightly higher than male response rate.

  4. Response was slightly higher when only one person was selected in the DU.

  5. Response was highest for non-MSA areas and lowest for large MSA areas.

  6. Response was higher in noncertainty PSUs than in certainty PSUs.

Similarly, the nonresponse categories for the field experiment and comparison sample exhibited similar patterns of nonresponse across subgroups. For example, Exhibit 10.1.5 shows how the percentage of eligible persons refusing to be interviewed tended to vary directly with the person's age. A similar pattern was evident, with a major difference in level.

Exhibit 10.1.6 shows an increasing rate of language barrier cases, by age groups, for both the field experiment and comparison surveys. The levels were similar for both surveys, as well as the pattern of nonresponse.

To summarize, the person-level response rate was 11.0 percentage points lower in the field experiment compared with the rate for the 1997 NHSDA Quarter 4 sample. The primary reason for the difference was the higher refusal rate experienced in the field experiment. Despite the lower level of response during the field experiment, the two surveys showed many similar response and nonresponse patterns. That is, except for the significantly higher field experiment refusal rate, the two surveys showed many similar patterns across both subgroups and categories of nonresponse.

Exhibits 10.1.7 and 10.1.8 contain the detailed person response rate information for 12 to 17 year olds and for adults 18 years of age or older. There are some interesting differences between the response rates and nonresponse patterns for these two age groups. Again, the refusal rates were much higher for the field experiment than for the comparison sample. The summary information in Exhibit 10.1.9 contrasts the results for 12 to 17 year olds with those for persons 18 years of age or older. Clearly, the lower field experiment response rates were primarily due to the field experiment's higher refusal rates for both age groups. The field experiment's nonresponse rates for both age groups also were higher than those for the comparison survey for the physical or mental problems and other nonresponse categories. The field experiment and comparison surveys both yielded approximately the same percentages of persons who could never be found athome, and this held for both age groups.

The language barrier information in Exhibit 10.1.9 is quite interesting. The same percentage of nonresponse (10.5%) was obtained in both surveys for adults, whereas for 12- to 17-year-old persons, the field experiment found fewer language barrier cases (1.2%) than did the comparison sample (4.5%). This is likely due to the coding of 1997 NHSDA Quarter 4 persons who responded with the Spanish PAPI instrument as language barriers for the purpose of this analysis. It appears that many of the 12- to 17-year-old sample members, who might have chosen to use the Spanish instrument in the comparison sample, responded instead in the field experiment to the English CAPI instrument because they were capable of responding in either language, and English was their only choice in the field experiment.

The detailed field experiment and comparison survey response rate tabulations for the two age groups show many of the same patterns of nonresponse across the subpopulations tabulated. Both age groups in both surveys exhibited the following person-level response patterns:

  1. Response was lower for Hispanics than for blacks or whites and others.

  2. Female response rate was slightly higher than male response rate.

  3. Response was slightly higher when only one person was selected in the DU.

  4. Response was highest for non-MSA areas and lowest for large MSA areas.

  5. Response was higher in noncertainty PSUs than in certainty PSUs.

To a considerable extent, several of these patterns result from the lack of Spanish-language CAPI instrumentation. This drastically lowered the computed person response rates for both age groups for Hispanics, most of whom lived in the West region, in large MSAs, and certainty PSUs.

Exhibit 10.1.10 shows for youths and adults the differences between the field experiment and 1997 NHSDA Quarter 4 response rates. This comparison of overall and subgroup response rates shows that both surveys obtained higher response rates for 12 to 17 year olds than for adults. The response rate for the younger age group was higher for both surveys overall and for each of the 16 subgroups examined.

The exhibit also shows that the negative differences between the field experiment and comparison group response rates were much larger for persons 18 years of age or older than they were for persons 12 to 17 years of age. This pattern held for the total samples, as well as for each of the 16 subgroups.

Exhibit 10.1.1 Dwelling Unit Screening Rate Percentages, by Region, MSA Size, and Certainty/Noncertainty Areas

 

Total Sample

Northeast

Midwest

South

West

 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Ineligible Dwelling Unit

16.28

15.30

21.71

17.37

13.68

14.89

17.20

16.26

12.96

12.01

    Vacant

11.80

11.32

16.07

11.06

11.29

11.71

12.04

12.52

8.56

8.81

    Not a Primary Residence

1.63

1.79

2.20

2.88

0.16

0.94

2.25

2.15

1.39

0.91

    Not a Dwelling Unit

2.30

2.07

3.04

3.43

1.97

2.10

2.04

1.45

2.74

2.14

    Other

0.56

0.12

0.40

0.01

0.26

0.13

0.87

0.14

0.26

0.14

Eligible Dwelling Unit, Not Screened for Eligible Persons

14.60

6.37

10.26

6.58

17.57

6.62

12.26

4.86

19.59

8.88

    No One at Home

2.08

2.09

2.80

2.98

1.72

1.96

1.86

1.79

2.55

2.05

    Refusal

5.84

2.46

5.04

2.65

7.01

2.47

5.33

1.94

6.26

3.29

    Denied Access

2.91

0.59

0.57

0.29

3.26

1.32

1.14

0.23

8.24

0.90

    Newton Screener Problem

0.57

0.00

0.32

0.00

0.52

0.00

0.64

0.00

0.60

0.00

    Other Nonresponse (Group Quarters)1

3.19

1.24

1.53

0.67

5.06

0.88

3.29

0.90

1.94

2.64

Eligible Dwelling Units, Screened for Eligible Persons

85.40

93.63

89.74

93.42

82.43

93.38

87.74

95.14

80.41

91.12

                     
 

Total Sample

Large MSA

Small MSA

Non-MSA

   
 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Ineligible Dwelling Unit

16.28

15.30

15.19

12.52

17.02

18.19

17.85

19.07

   

    Vacant

11.80

11.32

11.45

9.33

11.49

13.48

13.54

13.83

   

    Not a Primary Residence

1.63

1.79

0.62

0.81

3.36

2.73

0.70

3.33

   

    Not a Dwelling Unit

2.30

2.07

2.42

2.27

1.68

1.81

3.34

1.90

   

    Other

0.56

0.12

0.70

0.11

0.50

0.17

0.26

0.01

   

Eligible Dwelling Unit, Not Screened for Eligible Persons

14.60

6.37

17.36

7.92

12.05

4.91

11.99

3.48

   

    No One at Home

2.08

2.09

2.47

2.66

2.07

1.60

0.91

0.87

   

    Refusal

5.84

2.46

6.58

2.89

5.93

2.09

3.37

1.56

   

    Denied Access

2.91

0.59

5.54

0.96

0.64

0.18

0.05

0.03

   

    Newton Screener Problem

0.57

0.00

0.67

0.00

0.60

0.00

0.16

0.00

   

    Other Nonresponse (Group Quarters)1

3.19

1.24

2.09

1.40

2.81

0.73

7.49

1.02

   

Eligible Dwelling Units, Screened for Eligible Persons

85.40

93.63

82.64

92.08

87.95

95.55

88.01

96.52

   
                     
 

Total Sample

Certainty Area

Noncertainty Area

       
 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

       

Ineligible Dwelling Unit

16.28

15.30

15.77

12.74

16.80

18.76

       

    Vacant

11.80

11.32

11.96

9.39

11.63

13.85

       

    Not a Primary Residence

1.63

1.79

0.74

0.92

2.53

2.93

       

    Not a Dwelling Unit

2.30

2.07

2.37

2.30

2.22

1.76

       

    Other

0.56

0.12

0.70

0.12

0.42

0.11

       

Eligible Dwelling Unit, Not Screened for Eligible Persons

14.60

6.37

17.43

7.61

11.72

4.64

       

    No One at Home

2.08

2.09

2.43

2.48

1.73

1.53

       

    Refusal

5.84

2.46

6.61

2.71

5.07

2.11

       

    Denied Access

2.91

0.59

5.56

0.89

0.22

0.17

       

    Newton Screener Problem

0.57

0.00

0.65

0.00

0.49

0.00

       

    Other Nonresponse (Group Quarters)1

3.19

1.24

2.19

1.54

4.22

0.82

       

Eligible Dwelling Units, Screened for Eligible Persons

85.40

93.63

82.57

92.39

88.28

95.36

       

1The Newton application used for the 1997 field experiment did not handle group quarters.

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures: 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.2 Dwelling Unit Screening Nonresponse Categories: Field Experiment Versus NHSDA Quarter 4 Comparison Group

 

Field Experiment

Comparison Group

Difference

Small or No Difference

     

    No one home

2.1

2.1

 

    Newton screener problem

0.6

0.0

 

    Total

2.7

2.0

-0.7

       

Large Difference

     

    Refusal

5.8

2.5

 

    Denied access

2.9

0.6

 

    Other nonresponse1

3.2

1.2

 

    Total

11.9

4.3

-7.6

1Group quarters component was 1.4.

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.3 Person Response Rate Percentages for All Persons, by Sociodemographic Characteristics

 

Total Sample

12 to 17 Year Olds

18 to 25 Year Olds

26 to 34 Year Olds

35+ Year Olds

 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Nonresponse Rate

37.34

26.31

27.33

18.50

40.28

27.66

46.39

28.59

52.58

31.70

    No one home/R unavailable

5.69

6.43

4.55

4.45

9.02

8.37

7.03

6.83

4.66

6.53

    Physical/mental incompetent

1.80

0.95

1.37

0.65

0.60

0.43

1.33

0.53

4.33

2.13

    Language barrier

5.94

8.78

1.17

4.49

7.21

9.58

11.03

10.56

12.65

11.19

    Refusal

19.76

8.62

17.44

7.95

18.44

7.04

20.34

8.85

26.29

10.59

    Other

4.14

1.53

2.80

0.96

5.01

2.24

6.65

1.82

4.66

1.26

Overall Response Rate

62.66

74.10

72.67

81.50

59.72

72.34

53.61

71.41

47.42

68.30

 
 

Total Sample

Hispanic

Non-Hisp., Black

Non-Hisp., All Other Races

 
 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Nonresponse Rate

37.34

26.31

42.96

45.32

34.20

17.81

35.97

19.70

   

    No one home/R unavailable

5.69

6.43

4.25

6.21

7.19

7.71

5.68

5.80

   

    Physical/mental incompetent

1.80

0.95

0.49

0.64

2.60

0.85

2.09

1.20

   

    Language barrier

5.94

8.78

20.15

30.18

0.62

0.17

1.11

0.75

   

    Refusal

19.76

8.62

13.11

6.34

19.95

7.32

23.24

10.81

   

    Other

4.14

1.53

4.98

1.96

3.84

1.75

3.85

1.14

   

Overall Response Rate

62.66

74.10

57.04

54.68

65.80

82.19

64.03

80.30

   
 
 

Total Sample

Male

Female

       
 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

       

Nonresponse Rate

37.34

26.31

38.85

28.42

35.94

24.66

       

    No one home/R unavailable

5.69

6.43

5.87

7.34

5.53

5.72

       

    Physical/mental incompetent

1.80

0.95

1.85

0.99

1.76

0.91

       

    Language barrier

5.94

8.78

6.20

9.32

5.71

8.35

       

    Refusal

19.76

8.62

20.58

9.26

19.00

8.12

       

    Other

4.14

1.53

4.35

1.51

3.95

1.55

       

Overall Response Rate

62.66

74.10

61.15

71.58

64.06

75.34

       
 
 

Total Sample

1 Person Selected in DU

2 People Selected in DU

       
 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

       

Nonresponse Rate

37.34

26.31

36.02

23.82

38.74

29.98

       

    No one home/R unavailable

5.69

6.43

5.96

5.87

5.40

7.26

       

    Physical/mental incompetent

1.80

0.95

2.15

0.97

1.43

0.91

       

    Language barrier

5.94

8.78

4.92

7.82

7.03

10.19

       

    Refusal

19.76

8.62

19.11

7.80

20.44

9.83

       

    Other

4.14

1.53

3.87

1.36

4.43

1.79

       

Overall Response Rate

62.66

74.10

63.98

76.18

61.26

70.02

       
 
 

Total Sample

Northeast

Midwest

South

West

 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Nonresponse Rate

37.34

26.31

37.83

18.77

34.84

24.25

34.90

21.07

45.30

43.80

    No one home/R unavailable

5.69

6.43

6.86

8.07

6.38

6.83

5.58

5.39

4.47

7.17

    Physical/mental incompetent

1.80

0.95

1.65

0.83

1.87

1.18

2.38

0.86

0.48

1.00

    Language barrier

5.94

8.78

4.49

4.41

1.24

3.42

4.01

6.92

16.27

19.95

    Refusal

19.76

8.62

20.57

4.56

23.95

11.14

19.05

6.70

16.59

13.18

    Other

4.14

1.53

4.26

0.90

1.40

1.69

3.88

1.20

7.50

2.50

Overall Response Rate

62.66

74.10

62.17

81.23

65.16

75.75

65.10

78.93

54.70

56.20

 
 

Total Sample

Large MSA

Small MSA

Non-MSA

   
 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Nonresponse Rate

37.34

26.31

43.84

32.37

34.38

21.88

25.19

13.26

   

    No one home/R unavailable

5.69

6.43

6.83

7.57

4.70

5.74

4.62

3.68

   

    Physical/mental incompetent

1.80

.95

2.07

0.98

1.22

0.94

2.31

0.82

   

    Language barrier

5.94

8.78

8.84

12.52

4.44

5.49

0.96

1.89

   

    Refusal

19.76

8.62

21.02

9.53

20.28

8.45

15.00

5.61

   

    Other

4.14

1.53

5.09

1.77

3.74

1.26

2.31

1.25

   

Overall Response Rate

62.66

74.10

56.16

67.63

65.62

78.12

74.81

86.74

   
 
 

Total Sample

Certainty Area

Noncertainty Area

       
 

Field Experiment

Comparison Group1

Field Experiment

Comparison Group

Field Experiment

Comparison Group

       

Nonresponse Rate

37.34

26.31

43.40

33.09

30.79

16.36

 

    No one home/R unavailable

5.69

6.43

6.45

7.38

4.87

5.04

 

    Physical/mental incompetent

1.80

0.95

1.64

0.99

1.97

0.89

 

    Language barrier

5.94

8.78

9.92

13.75

1.64

1.48

 

    Refusal

19.76

8.62

20.15

9.23

19.34

7.73

 

    Other

4.14

1.53

5.23

1.75

2.96

1.21

 

Overall Response Rate

62.66

74.10

56.60

66.91

69.21

83.64

 

1The response rates for the comparison group that are shown in this exhibit are slightly lower than those shown in Exhibit 5.4.2 (74.1% vs. 75.6%) because this analysis was done prior to finalizing the 1997 NHSDA screening and interviewing results. This difference does not change the conclusions.

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures: 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.4 Comparison of Person Nonresponse Categories for All Persons: Field Experiment Versus Quarter 4 Surveys

 

Field Experiment

Comparison Group

Difference

Small or No Difference

     

    Not home/response unavailable

5.7

6.4

 

    Physical/mental incapable

1.8

1.0

 

    Language barrier

5.9

8.81

 

    Total

13.4

16.2

+2.8

       

Large Difference

     

    Refusal

19.8

8.6

 

    Other nonresponse

4.1

1.5

 

    Total

23.9

10.1

-13.8

1Includes respondents who completed a Spanish-language questionnaire.

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.5 Refusal Rates, by Age Group and Sample

Sample

12 to 17 Years

18 to 25 Years

26 to 34 Years

35 Years or Older

Field Experiment

17.4

18.4

20.3

26.3

Comparison Group

8.0

7.0

8.8

10.6

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.6 Language Barrier Rates, by Age Group and Sample

Sample

12 to 17 Years

18 to 25 Years

26 to 34 Years

35 Years or Older

Field Experiment

1.2

7.2

11.0

12.6

Comparison Group

4.5

9.6

10.6

11.2

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.7 Person Response Rate Percentages for 12- to 17-Year-Old Youths, by Race/Ethnicity, Gender, and Number of Sample Persons in DU

 

Total Sample

Hispanic

Non-Hisp., Black

Non-Hisp., All Other Races

 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Nonresponse Rate

27.33

18.50

24.24

26.33

26.63

14.91

29.35

15.48

No one home/R unavailable

4.55

4.45

3.73

4.30

5.62

6.56

4.55

3.36

Physical/mental incompetent

1.37

0.65

0.47

0.76

0.59

0.72

2.21

0.53

Language barrier

1.17

4.49

3.26

14.75

0.00

0.07

0.52

0.37

Refusal

17.44

7.95

13.29

5.38

17.75

6.27

19.61

10.57

Other

2.80

0.96

3.50

1.14

2.66

1.30

2.47

0.66

Overall Response Rate

72.67

81.50

75.76

73.67

73.37

85.09

70.65

84.52

 
 

Total Sample

Male

Female

   
 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Nonresponse Rate

27.33

18.50

26.48

19.14

28.19

17.89

   

No one home/R unavailable

4.55

4.45

4.37

4.30

4.74

4.61

   

Physical/mental incompetent

1.37

0.65

1.67

0.79

1.05

0.51

   

Language barrier

1.17

4.49

1.29

4.75

1.05

4.25

   

Refusal

17.44

7.95

16.32

8.59

18.58

7.33

   

Other

2.80

0.96

2.83

0.72

2.77

1.20

   

Overall Response Rate

72.67

81.50

73.52

80.86

71.81

82.11

   
 
 

Total Sample

1 Person Selected in DU

2 People Selected in DU

   
 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Nonresponse Rate

27.33

18.50

26.10

15.81

28.55

20.80

   

No one home/R unavailable

4.55

4.45

3.38

4.01

5.74

4.83

   

Physical/mental incompetent

1.37

0.65

1.56

0.64

1.17

0.65

   

Language barrier

1.17

4.49

0.91

3.05

1.43

5.72

   

Refusal

17.44

7.95

18.05

7.54

16.82

8.29

   

Other

2.80

0.96

2.21

0.56

3.39

1.30

   

Overall Response Rate

72.67

81.50

73.90

84.19

71.45

79.20

   

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures: 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.8 Person Response Rate Percentages for Adults 18 Years Old or Older, by Race/Ethnicity, Gender, and Number of Sample Persons in DU

 

Total Sample

Hispanic

Non-Hisp., Black

Non-Hisp., All Other Races

 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Nonresponse Rate

46.80

29.39

63.29

53.36

39.66

18.86

42.65

21.36

No one home/R unavailable

6.77

7.21

4.81

7.01

8.32

8.13

6.82

6.77

Physical/mental incompetent

2.21

1.06

0.51

0.59

4.05

0.89

1.97

1.46

Language barrier

10.46

10.47

38.48

36.71

1.07

0.21

1.71

0.91

Refusal

21.96

8.88

12.91

6.75

21.54

7.71

26.90

10.90

Other

5.41

1.76

6.58

2.30

4.69

1.92

5.25

1.33

Overall Response Rate

53.20

70.61

36.71

46.64

60.34

81.14

57.35

78.64

 
 

Total Sample

Male

Female

   
 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Nonresponse Rate

46.80

29.39

51.90

32.71

42.57

26.99

   

No one home/R unavailable

6.77

7.21

7.45

8.75

6.19

6.11

   

Physical/mental incompetent

2.21

1.06

2.03

1.08

2.36

1.05

   

Language barrier

10.46

10.47

11.38

11.43

9.68

9.77

   

Refusal

21.96

8.88

25.07

9.57

19.37

8.39

   

Other

5.41

1.76

5.96

1.88

4.95

1.67

   

Overall Response Rate

53.20

70.61

48.10

67.29

57.43

73.01

   
 
 

Total Sample

1 Person Selected in DU

2 People Selected in DU

   
 

Field Experiment

Comparison Group

Field Experiment

Comparison Group

Field Experiment

Comparison Group

   

Nonresponse Rate

46.80

29.39

44.92

26.06

48.89

35.55

   

No one home/R unavailable

6.77

7.21

8.28

6.39

5.07

8.74

   

Physical/mental incompetent

2.21

1.06

2.68

1.07

1.69

1.06

   

Language barrier

10.46

10.47

8.52

9.15

12.61

12.91

   

Refusal

21.96

8.88

20.07

7.87

24.06

10.76

   

Other

5.41

1.76

5.37

1.58

5.46

2.08

   

Overall Response Rate

53.20

70.61

55.08

73.94

51.11

64.45

 

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures: 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.9 Percentage of Eligible Persons, by Response Category and by Age Group

Age Group and Survey

Response Not Available

Physical or Mental Problems

Language Barrier

Refusal

Other
Nonresponse

Response
Rate

12 to 17 Year Olds

           

Field experiment

4.6

1.4

1.2

17.4

2.8

72.7

Comparison group

4.4

0.6

4.5

8.0

1.0

81.5

Difference

0.2

0.8

-3.3

9.0

1.8

-8.8

Adults

           

Field experiment

6.8

2.2

10.5

22.0

5.4

53.2

Comparison group

7.2

1.1

10.5

8.9

1.8

70.6

Difference

-0.4

1.1

0.0

13.1

3.6

-17.4

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

Exhibit 10.1.10 Field Experiment Versus Comparison Group Response Rates, by Subgroup

 

Response Rates for
12 to 17 Year Olds

 

Response Rates for
18 Year Olds or Older

Subgroup

Field Experiment

Comparison Group

Difference

Field Experiment

Comparison Group

Difference

Total

72.7

81.5

-8.8

 

53.2

70.6

-17.4

             

Hispanic

75.8

73.7

+2.1

36.7

46.6

-9.9

Black

73.4

85.1

-11.7

60.3

81.1

-20.8

White/Other

70.5

84.8

-14.0

57.4

78.6

-21.2

             

Males

73.5

80.9

-7.4

48.1

67.3

-19.2

Females

71.8

82.1

-10.3

57.4

73.0

-15.6

             

1 Selected in DU

73.9

84.2

-10.3

55.1

73.9

-18.8

2 Selected in DU

71.4

79.4

-8.0

51.1

64.4

-13.3

             

Northeast

70.0

86.3

-16.3

53.9

79.4

-25.5

Midwest

69.4

80.9

-11.0

61.3

73.6

-12.3

South

75.7

86.3

-10.6

55.6

76.0

-20.4

West

71.1

68.9

+2.2

38.1

51.2

-13.1

             

Large MSAs

67.9

77.1

-9.2

46.5

64.1

-17.6

Small MSAs

75.0

84.2

-9.2

55.8

75.6

-19.8

Non-MSA areas

79.0

90.2

-11.2

69.7

85.2

-15.5

             

Certainty PSUs

69.6

77.1

-7.5

45.4

63.1

-17.7

Noncertainty PSUs

75.7

87.4

-11.7

62.4

82.0

-19.6

DU = dwelling unit; MSA = metropolitan statistical area; PSU = primary sampling unit.

Sources: National Household Survey on Drug Abuse: Development of Computer-Assisted Interviewing Procedures; 1997 Field Experiment. 1997 National Household Survey on Drug Abuse: Quarter 4.

10.2 Effect of Electronic Instruments on Response Rates

Why were the field experiment's response rates so much lower than those from the comparison group sample? Was it because the field experiment used electronic survey instruments? It is not possible to answer this question directly, due to the confounding of the effect of the separate survey staffs with that of the electronic versus PAPI instrumentation. Still, several data and anecdotal indications may partially answer the question. We discuss these indications in this section.

There were some problems with the electronic instruments and the case management system (CMS). The Newton screener and the CAPI instrument were newly developed applications that, due to time limitations, had not been perfected by the beginning of the field experiment's data collection. Due to timing, the FI training also was less comprehensive than a fully developed, ideal training package. Field management staff noted that the Newton worked very well with only minor problems and that the computer-assisted personal interviewing (CAPI) software also worked quite well. Minor problems were encountered in changing some of the previous entries, and this may have contributed to the higher field experiment nonresponse rates.

Significant problems with the CMS were encountered early in the survey quarter. The FSs had difficulty transferring cases among interviewers; this was perceived as a major problem with CAPI cases, although it was not a problem with Newton cases. Headquarters and field supervisory staff also became very frustrated with the lack of accuracy in the CMS field reports, which they were using to manage the survey. The fixed time interval in which to complete the fieldwork and the frustrating problems experienced during the early data collection period may have pushed a larger fraction of the data collection into November and December 1997 and these holiday months are less conducive to effective fieldwork. Thus, such factors as the holiday months, the less than optimal flexibility of the newly developed electronic survey application, especially the CMS, and a new field staff probably increased field experiment's nonresponse.

How much of the response rate differences could be due to using the electronic instruments? At the DU screening stage, the interviewers indicated through a special code that less than 1% of the eligible DUs were not screened because of Newton problems. This indicates that there was little if any DU nonresponse because the Newtons were used for screening. The nonresponse categories that showed large differences at the screening level were refusals, denied access, and other nonresponse (Exhibit 10.1.2). It is very difficult to believe that interviewers approaching households would be more likely to be denied access to DUs, or that more sample households would be more apt to refuse to give the DU screening information because the FI carried a Newton computer. Experienced supervisors and interviewers, however, thought that there could have been another effect due to the electronic screener. Interviewers know that people are not selected in most households, and if the household has no youths and no Hispanics, that there is a low chance that anyone will be selected. When using the paper sample selection tables, interviewers can determine if anyone will be selected for certain household types prior to the screening. They could then use this information to facilitate the screening by stating, for example, "I just have a very few questions. I don't think that anyone who lives here is eligible for this survey, but I just need to ask you a few questions to be sure." This option is not available to the interviewers using the electronic screening application because the sample selection tables are not visible to the interviewer. Although this is a possibility, it is likely to be a less important factor than the lack of expertise on the part of the interviewers.

Exhibit 10.1.4 indicates that the refusal and other nonresponse categories accounted for the lower person-level response rates for the field experiment. It also is difficult to believe that more persons would refuse to participate in the survey because the interviewer planned to use a portable computer to complete portions of the interview. Most refusals occur before interviewsare begun; thus, many of the sample persons who refused may not have even been aware that the interview would involve using a computer.

Higher nonresponse in the other nonresponse category could indicate difficulties due to electronic data collection. As discussed earlier in this section, there is reason to believe that the deficiencies in the newly developed electronic survey application may have contributed to the higher field experiment nonresponse. Use of the other nonresponse category in the field experiment accounted for approximately 24% of the response rate decrease, at both the DU screening and person interviewing levels.

Therefore, although no definite conclusions can be made in this regard, it is reasonable to attribute at most 20% to 25% of the field experiment's response rate shortfall to the electronic survey environment. The deficiencies in the electronic survey application, as discussed earlier in this section, were the lack of a Spanish translation, difficulties in changing some entries in the computers, and problems in the CMS. Fortunately, all of these deficiencies can be corrected, avoiding any such response rate decreases due to the electronic survey environment in future NHSDA survey years.

10.3 Effect of Experimental Environment on Response Rates

The experimental environment in which the field experiment and comparison surveys were conducted differed in several ways. First, a completely separate staff of FSs and FIs were hired to work on the field experiment, which ensured that the 1997 NHSDA could not be negatively affected in that none of its data collection resources were diverted to conducting the field experiment survey. The field experiment's FSs were less experienced than the comparison survey's FSs. And the field experiment's supervisors also lacked the specific experience of currently working as 1997 NHSDA FSs.

Similar statements can be made in comparing the 165 FIs who worked on the pretest with the 270 FIs who worked on the comparison survey. The two groups of interviewers were remarkably similar on many of the characteristics for which data were available, including their educational level, gender, age, race, and ethnicity. However, the comparison group interviewers were more experienced in general survey interviewing and were all currently working as 1997 NHSDA FIs. Headquarters staff also commented that the field experiment's interviewers as a group lacked the level of refusal conversion expertise of the interviewers working on the comparison survey. This is obviously a critical skill for gaining access to sample DUs and convincing reluctant individuals that they should cooperate with the survey.

The two field staffs were both responsible to the same headquarters survey management staff. The central office, FSs, and FIs all exhibited very positive attitudes toward the field experiment study. Still, there may have been some unavoidable tendency to see the field experiment as a onetime add-on to the continuing NHSDA "bread and butter" project. There are no data to suggest that there was any effect on the study response rates due to the attitudes of any project staff.

Returning to the interviewing experience deficiency of the field experiment interviewers, it may be observed in Exhibits 10.1.2 and 10.1.4 that the categories of both DU screening and person interviewing that accounted for small response rate differences are the categories that one would not expect to be affected by interviewer experience levels. That is, one would not expect interviewer experience or expertise to have a significant effect on the percentage of dwellings in which no one was found at home, the eligible persons who were unavailable when calls were made, who were physically or mentally unable to respond to the survey, or who were unable to participate due to language capability.

Conversely, the categories of nonresponse that are logically related to large response rate differences in Exhibits 10.1.2 and 10.1.4 are those that are probably related to interviewer experience levels (refusals, denied access, and other nonresponse).

In addition, anecdotal information indicates that the traveling interviewer strategy used for the field experiment was not as effective as expected. Interviewers selected for this assignment were experienced veteran interviewers but lacked computer expertise and were not accustomed to working for extended periods away from their homes. Once the traveling interviewer team was assembled and trained, they had difficulty working effectively due to the remaining data collection time and complications of the holiday season.

No conclusive experimental data clearly show whether the experimental environment of the field experiment was responsible for the lower response rates. But it appears that the lower level of general survey experience and specific NHSDA experience, plus the refusal conversion experience deficit of the field experiment's interviewing staff, may have been the main factors causing the response rate shortfall. Thus, it seems reasonable to conclude that roughly 75% of the shortfall was due to the experimental environment.

10.4 Effect of Electronic Instruments on Respondent Attitudes and Self-Reported Willingness to Respond

Chapter 6 contains a detailed examination of the field experiment (CAPI) and comparison group (PAPI) respondents' attitudes toward the surveys. This analysis, based on respondent debriefings conducted following both surveys, concluded the following:

  1. The CAPI respondents were more likely to feel that the interviewers did not see any of their answers.

  2. With respect to protecting their privacy, only about 11% of the respondents to both surveys felt that answer sheets were the best method of data collection.

  3. The CAPI survey respondents found the survey somewhat more interesting than the PAPI respondents did.

  4. A slightly higher percentage of the CAPI respondents said they would be likely to participate in the survey again.

  5. The CAPI respondents were slightly more likely to feel that their survey information would never be linked with their names.

Thus, the respondent debriefings confirmed the results of the 1996 feasibility experiment: The CAPI respondents tended to be more positive than PAPI respondents on privacy and confidentiality issues; they preferred to use the computer to respond to the survey; and those responding by computer said they would be more likely to participate in the survey if it were done again. All of these results seem to indicate respondent attitudes toward the electronic survey methods were very positive, and that there is no reason to believe that the electronic survey environment will have any negative effects on respondents' willingness to completely respond to future NHSDAs once they have begun the interview. However, most decisions to respond are made prior to the subject encountering the instrument.

Top Of PageTable of Contents

This page was last updated on June 16, 2008.