Chapter 8. Population, Sampling and Sample-Size Planning
8.1 Define the population precisely
Specify the target population, accessible population, unit of analysis, inclusion and exclusion criteria, setting, period and sampling frame. The unit may be an employee, customer, manager, organisation, branch, transaction or country. Confusing individual-level and organisation-level variables can create ecological or atomistic errors.
8.2 Probability and non-probability sampling
| Method | Description | Advantages | Risks/requirements |
|---|---|---|---|
| Simple random | Each frame unit has a known, equal selection chance | Clear probability basis | Needs a suitable complete frame |
| Systematic | Select every kth unit after a random start | Operationally simple | Periodicity in the list can bias selection |
| Stratified | Sample within meaningful strata | Can improve representation/precision | Requires stratum data and correct weighting |
| Cluster | Sample groups then units within them | Cost-efficient for dispersed populations | Design effect and cluster-level dependence |
| Multistage | Selection occurs in multiple stages | Practical for complex populations | Analysis must reflect stages and probabilities |
| Convenience | Recruit readily available cases | Fast and inexpensive | Selection bias; limited population inference |
| Purposive/criterion-based | Recruit cases meeting defined criteria | Useful for specialised populations | Inclusion probabilities often unknown |
| Quota | Recruit to fill category targets | May improve visible composition | Does not eliminate selection bias |
| Snowball/referral | Participants refer others | Can reach hidden networks | Network dependence and strong selection effects |
8.3 Sample size is not one magic number
Sample size depends on the estimand, design, expected effect or precision, variability, significance level, desired power, number of predictors, clustering, repeated measures, missingness, response rate and intended subgroup analyses. Rules such as “ten cases per variable” are not universal justifications. A sample-size calculation should be tied to the primary outcome and primary analysis.
8.4 Common planning approaches
- Precision-based estimation: choose a confidence level and acceptable margin of error for a proportion or mean.
- Power analysis: define a meaningful effect size, alpha level, desired power (often 80% or 90%), test and design.
- Regression planning: consider expected model fit, number of parameters, predictor distributions, collinearity and validation needs.
- Cluster designs: inflate for design effect, often approximated as 1 + (average cluster size − 1) × intraclass correlation, when assumptions fit.
- Attrition/nonresponse: recruit above the analyzable target using a defensible response/retention assumption.
- Complex models: use simulation or specialist guidance when analytic formulas are inadequate.
A response rate alone does not determine bias: a low response rate can sometimes produce limited bias and a high response rate can still be biased. Compare respondents and nonrespondents on available frame variables, document follow-up procedures, and describe the limits of inference.
8.5 Illustrative finite-population formula
For a simple random sample estimating a proportion under assumptions, a common starting formula is n₀ = Z² p(1−p) / e², where Z is the normal critical value for the chosen confidence level, p is the anticipated proportion (0.50 is conservative when unknown), and e is the desired margin of error. For a known finite population N, one adjustment is n = n₀ / [1 + (n₀−1)/N]. This is not appropriate for every design and does not replace power analysis for hypothesis tests.
8.6 Report sampling transparently
State how the frame was obtained, how cases were selected, who was eligible, how many were approached, how many responded, how exclusions were applied, how weights were used, and what population the analysis can reasonably represent. For convenience samples, describe the sample as the achieved sample and be cautious about population-level generalisation.