Prof. Dr. Larry AdamsAcademic, Author & Researcher
How to Conduct Quantitative Research · Chapter 18 of 33

Part III — DATA ANALYSIS AND INTERPRETATION

Chapter 14. Inferential Statistics: Choosing and Running the Right Test

14.1 The test follows the question and design

Do not choose a test because it is the one most familiar or the one that produces significance. Decide the estimand, unit of analysis, outcome scale, number of groups, independence structure, design, assumptions and covariates. For each analysis, state the estimate, uncertainty, test statistic where useful, degrees of freedom when relevant, p-value and effect size.

Research purpose/data situationCommon methodImportant checks/alternatives
One sample mean vs benchmarkOne-sample t-testIndependence, distribution of mean; robust/bootstrap alternatives
Two independent meansIndependent-samples t-test / Welch t-testIndependence, outliers; Welch handles unequal variances
Paired measurementsPaired t-testPairing, distribution of differences
Three or more group meansANOVA / Welch ANOVAIndependence, residuals, variance; planned contrasts/post-hoc control
Ordinal or strongly non-normal comparisonMann–Whitney, Wilcoxon, Kruskal–WallisThese test distribution/rank differences, not automatically medians
Two categorical variablesChi-square test / Fisher exactExpected cell counts, independent observations
Two continuous variablesPearson correlationLinearity, influential cases; Spearman for monotonic rank association
Binary outcomeLogistic regressionIndependence, functional form, separation, calibration
Count outcomePoisson/negative binomial regressionExposure, overdispersion, zero patterns
Time-to-event outcomeSurvival analysisCensoring, proportional hazards if using Cox model
Repeated/nested dataMixed models / GEEWithin-cluster dependence and covariance structure

14.2 Assumptions are model-specific

Normality of raw variables is not a universal requirement for every parametric test. For linear models, assumptions concern the conditional mean structure, residual behaviour, independence, variance and specification. Large samples do not excuse dependence or severe model misspecification. Use graphical diagnostics alongside tests, because formal tests can be too sensitive in large samples and underpowered in small ones.

14.3 Statistical significance versus practical importance

A small p-value indicates that the observed data would be relatively incompatible with a specified null model under its assumptions. It does not measure the size or importance of a relationship, the probability that the hypothesis is true, or the chance that results are due to bias. A non-significant result does not prove no effect; it may reflect imprecision. Interpret effect sizes, confidence intervals, prior evidence, study design and practical thresholds together.

14.4 Multiple testing

Testing many outcomes, subgroups or model specifications increases the chance of at least one false-positive result. Prespecify primary outcomes and contrasts. Depending on the purpose, consider family-wise error control (such as Holm) or false-discovery-rate control (such as Benjamini–Hochberg). Clearly label exploratory analyses and avoid presenting only the most favourable results.

14.5 A generic reporting template

“The estimated association between X and Y was [estimate] (95% CI [lower, upper], p = [value]), after adjustment for [prespecified covariates]. The magnitude was [interpretation in the study context]. The estimate is compatible with [range of plausible values]; because the study was [design], the finding should be interpreted as [association/prediction/causal estimate only if justified].” Replace bracketed fields with actual results and never invent values.