Statistical Power Analysis
Perhaps the most frequently asked question that we receive is, “What sample size do I need?”. The answer to this question is influenced by a number of factors, including the purpose of the study, population size, the risk of selecting a “bad” sample, and the allowable sampling error.
In order to best estimate the sample size that you’d require or to validate that the sample you’ve planned is sufficient, we would conduct a power analysis for your study. To do this, we’d either refer to any pilot data you may have collected (the best choice), or to parameters in the relevant literature (if you’ve not done a pilot test). If you have a very esoteric study and neither of those options is reasonable, then we would use “rule of thumb” values for the effect sizes.
There are three major considerations when doing a power analysis for sample size determination, and we will consider all of these things while doing your power analysis:
- The general approach to determining sample size assumes that a simple random sample is the sampling design. More complex designs, e.g., stratified random samples, must take into account the variances of subpopulations, strata, or clusters before an estimate of the variability in the population as a whole can be made.
- The sample size should be appropriate for the analysis that is planned. If descriptive statistics are to be used, for example, then any reasonable sample size will suffice. On the other hand, a good size sample, approximately n=150+, is usually needed for multiple regression, analysis of covariance, or log-linear analysis, which might be performed for more rigorous state impact evaluations. In addition, an adjustment in the sample size may be needed to accommodate a comparative analysis of subgroups (e.g., such as an evaluation of program participants with nonparticipants). Also, skewed distributions can result in serious departures from normality even for moderate size samples, causing the need for a larger sample.
- Finally, the sample size formulas provide the number of responses that need to be obtained. Many researchers commonly add 25%+ to the planned sample size to compensate for persons that the researcher must remove from the sample for some reason. The number of mailed surveys or planned interviews also can be substantially larger than the number required, based on the assumed response rate.
We also can conduct power analyses on more complex psychometric studies using structural equation modeling. For this type of study, we would need details on how you plan to set up the SEM (latent variables, their indicators, their correlations, etc.). We then conduct the power analysis based on the methods described in McCallum, Browne, Sugawara (1996, also cited in the Klein’s SEM text ‘Principles and Practices of Structural Equation Modeling’). This method involves performing a custom SAS routine for calculating power and sample size needed in order to accept or reject the null hypothesis that the model is a good fit of the data, based on the Root Mean Square Error of Approximation (RMSEA) metric.