Statistical Significance of Differences

This section describes the methods that were used to compare prevalence estimates. Customarily, the observed difference between estimates is evaluated in terms of its statistical significance. "Statistical significance" refers to the probability that a difference as large as that observed would occur due to random error in the estimates if there were no difference in the prevalence rates for the population groups being compared. In Chapter 2, comparisons were made between estimates in the 1996 NHSDA survey and the 1997 survey and in later chapters between estimates within the 1997 survey. The significance of observed differences is reported in Chapter 2 at the 0.05 and 0.01 levels, and in later chapters at the 0.05 level. However, the reader may wish to compare prevalence estimates from two groups for which the significance of the difference is not reported.

To compare the prevalence of drug use for 1996 versus 1997, one can test the hypothesis of no difference in prevalence rates using the standard difference in proportions test, expressed as

where

p1 = 1996 estimated proportion,
var(p1) = variance estimate for p1,
p2 = 1997 estimated proportion,
var(p2) = variance estimate for p2, and
cov(p1,p2) = covariance between p1 and p2.

Under the null hypothesis of no difference in prevalence rates, Z is asymptotically distributed as a normal random variable; calculated values of Z can, therefore, be referred to the unit normal distribution to determine the corresponding probability level (i.e., p value). Because the 1996 and 1997 NHSDAs used a high percentage of overlapping sample segments (see Appendix D), the covariance term in the formula for Z is greater than zero in comparisons of the 1996 and 1997 surveys.

For comparing prevalence estimates within the same survey, the same Z statistic quoted above can be used. The covariance term in the formula for Z is again greater than zero because of the small positive covariance expected between any two nonindependent prevalence estimates.

This page was last updated on December 30, 2008.