When Does It Matter? | GLM Explainer

The decision guide

Click any row for the reasoning and a pointer to where this site demonstrates it.

SITUATIONPARAMETRIC ROUTECOMPUTATIONAL ROUTELEAN

Large n, model assumptions met fast, familiar, exact enough slower, same answer parametric

When the quadratic approximation holds, the Wald machinery is unimprovable in practice, and every reader knows how to interpret it. The bootstrap handshake on the heart-rate slope showed the two routes matching to two decimal places — agreement you can bank, and a reason not to burn compute. Report summary().

Small n (< 30–50) SEs may mislead honest about asymmetry bootstrap + check

Small samples are where surfaces go non-quadratic and Wald intervals go symmetric anyway. The three-route comparison's logistic scenario ($n = 30$) shows the bootstrap and the posterior agreeing against Wald. Run the bootstrap; if it splits from Wald, trust the split — and consider the walked posterior as referee.

Skewed data, quantile summaries no formula to use works unchanged bootstrap (only option)

Medians, deciles, ratios of quantiles, trimmed means, gaps between group medians — anything you can compute, the loop can interval. The blood-pressure demo put honest error bars on five different percentiles with one identical recipe. There is no parametric column to compare against; that is the point.

Testing “no effect” (β = 0) Wald or LR test permutation test either — agree usually

For decent samples the Wald, likelihood-ratio and permutation answers coincide (the slope permutation rebuilt the analytic SE almost exactly). The permutation test earns its keep with small groups, odd statistics, or audiences who distrust distributional assumptions — a p-value that is literally a count is hard to argue with.

Parameter near a boundary intervals leak out of range in range but granular Bayesian, bootstrap 2nd

The 2-events-in-20 scenario on Three Routes: Wald crosses zero, the bootstrap snaps to a 21-point lattice, the flat-prior posterior glides smoothly inside $(0, 1)$. Boundaries are the sampling route's home turf; the bootstrap is the respectable second choice.

Model comparison / selection AIC/BIC cross-validation either (related!)

AIC approximates out-of-sample prediction error analytically; cross-validation estimates the same thing computationally — the meat-brain and silicon-brain answers to one question (asymptotically, leave-one-out CV and AIC even select the same models). The same duality this whole section is built on, one level up.

What resampling cannot fix

A biased sample, faithfully rebiased. The bootstrap's licence — the sample stands in for the population — is also its only demand, and when it fails, resampling fails silently. If your data over-represent the young, every resample over-represents the young with beautiful precision, and your immaculately bootstrapped confidence interval brackets the wrong number. No amount of silicon fixes representativeness by resampling alone.

Exchangeability, for permutation tests. Shuffling assumes that under the null, labels are arbitrary — observations are exchangeable. Clustered, paired or time-ordered data violate this: shuffle within clusters or blocks, or the null distribution you build answers a question nobody asked.

The first failure has a repair, and it is the taxonomy's third branch: reweight the sample to known population totals. That is post-stratification, next — including the study that turned a wildly unrepresentative Xbox poll into an accurate election forecast.

House rule. When in doubt, run both routes. If they agree, report the parametric result (more familiar) with a clear conscience. If they disagree, the disagreement is a finding: something in the assumptions is broken, and reading which routes cluster usually tells you what.

Companion reading on JonStats: the Hacker Stats course closes with the same argument — resampling as both the alpha (intuition-builder) and omega (last resort) of statistics — and demonstrates the tidy infer pipeline for doing all of this declaratively in R.