Chapter 14. Inferential Statistics: Choosing and Running the Right Test
14.1 The test follows the question and design
Do not choose a test because it is the one most familiar or the one that produces significance. Decide the estimand, unit of analysis, outcome scale, number of groups, independence structure, design, assumptions and covariates. For each analysis, state the estimate, uncertainty, test statistic where useful, degrees of freedom when relevant, p-value and effect size.
| Research purpose/data situation | Common method | Important checks/alternatives |
|---|---|---|
| One sample mean vs benchmark | One-sample t-test | Independence, distribution of mean; robust/bootstrap alternatives |
| Two independent means | Independent-samples t-test / Welch t-test | Independence, outliers; Welch handles unequal variances |
| Paired measurements | Paired t-test | Pairing, distribution of differences |
| Three or more group means | ANOVA / Welch ANOVA | Independence, residuals, variance; planned contrasts/post-hoc control |
| Ordinal or strongly non-normal comparison | Mann–Whitney, Wilcoxon, Kruskal–Wallis | These test distribution/rank differences, not automatically medians |
| Two categorical variables | Chi-square test / Fisher exact | Expected cell counts, independent observations |
| Two continuous variables | Pearson correlation | Linearity, influential cases; Spearman for monotonic rank association |
| Binary outcome | Logistic regression | Independence, functional form, separation, calibration |
| Count outcome | Poisson/negative binomial regression | Exposure, overdispersion, zero patterns |
| Time-to-event outcome | Survival analysis | Censoring, proportional hazards if using Cox model |
| Repeated/nested data | Mixed models / GEE | Within-cluster dependence and covariance structure |
14.2 Assumptions are model-specific
Normality of raw variables is not a universal requirement for every parametric test. For linear models, assumptions concern the conditional mean structure, residual behaviour, independence, variance and specification. Large samples do not excuse dependence or severe model misspecification. Use graphical diagnostics alongside tests, because formal tests can be too sensitive in large samples and underpowered in small ones.
14.3 Statistical significance versus practical importance
A small p-value indicates that the observed data would be relatively incompatible with a specified null model under its assumptions. It does not measure the size or importance of a relationship, the probability that the hypothesis is true, or the chance that results are due to bias. A non-significant result does not prove no effect; it may reflect imprecision. Interpret effect sizes, confidence intervals, prior evidence, study design and practical thresholds together.
14.4 Multiple testing
Testing many outcomes, subgroups or model specifications increases the chance of at least one false-positive result. Prespecify primary outcomes and contrasts. Depending on the purpose, consider family-wise error control (such as Holm) or false-discovery-rate control (such as Benjamini–Hochberg). Clearly label exploratory analyses and avoid presenting only the most favourable results.
14.5 A generic reporting template
“The estimated association between X and Y was [estimate] (95% CI [lower, upper], p = [value]), after adjustment for [prespecified covariates]. The magnitude was [interpretation in the study context]. The estimate is compatible with [range of plausible values]; because the study was [design], the finding should be interpreted as [association/prediction/causal estimate only if justified].” Replace bracketed fields with actual results and never invent values.