Hacker Stats: Inference from a For-Loop

No formulas, no tables — resample the data you already have.

Where's the standard error of a median?

The inference section built uncertainty from calculus: a likelihood surface, its curvature at the peak, a Hessian to invert. That machinery is powerful — and strangely picky about what it will serve. Ask it for the standard error of a median and it has nothing to offer: no likelihood, no surface, no second derivative. The summary() table has no row for it.

Analytic answers do exist, technically — the asymptotic variance of a median involves $1/\left[2f(m)\sqrt{n}\right]$, where $f(m)$ is the density of the data at the median… which you would have to estimate, fragilely, before you could use it. The moment your statistic steps off the well-trodden path (means, GLM coefficients), each new case needs its own bespoke derivation — or gives up entirely.

The hacker's move. Skip the derivation. If you can compute the statistic once, you can compute it a thousand times on a thousand resampled versions of your data — and the spread of those thousand answers is the uncertainty. One recipe, any statistic. All it costs is silicon.

Meat brain vs silicon brain

The JonStats Hacker Stats course frames the trade honestly: resampling methods substitute silicon-brain effort (computation) for meat-brain effort (deriving and recalling analytic solutions). They are theoretically thin but deep — each rests on one simple, strong idea rather than a stack of distributional theory — and they are approximate, stochastic and general where classical recipes are precise, deterministic and specialist.

When Bradley Efron formalised the bootstrap in 1979, other names considered for it reportedly included Swiss Army Knife, Meat Axe and Shotgun. The first of those captures it best: not the sharpest tool for any single job, but the one that handles nearly every job you will actually meet.

JonStats argues these methods could be both the alpha of statistics (the clearest way to build intuition about null distributions and sampling variation, with no algebra in the way) and its omega (the only practical answer when no analytic solution exists). This section aims to show both ends.

Interactive: the loop, calibrated then unleashed

Below are the first 20 resting heart rates from this site's heart-rate study. Resample them with replacement: some values get picked twice or three times (badges), others not at all (faded). Each resample's statistic drops onto the histogram. Start with the mean — the textbook curve $\text{SE} = s/\sqrt{n}$ is drawn for comparison, so you can watch the loop rediscover it. Then switch to the median: the textbook curve vanishes, and the loop keeps working.

SAMPLE MEAN
RESAMPLES SO FAR
0
BOOTSTRAP SE*
TEXTBOOK SE = s/√n

The bootstrap principle. The sample stands in for the population, so the resample stands in for the sample: variation between resamples mimics the variation you would have seen between repeated real studies. For the mean, the loop's answer converges on the formula's (run all 1,000: SE* $\approx 1.26$ against the textbook $1.27$) — that agreement is what licenses trusting the loop where no formula exists.

The family tree

One idea, three methods, distinguished by two questions about how you resample (after the taxonomy in the JonStats course):

Sample data
Resample every row with equal probability?
YES — then…
With replacement?
NO — weight by population
Post-stratification repair a biased sample

How this section relates to the rest of the site

The curvature route reached the sampling distribution by calculus; this section reaches it by loop. Where both apply, they agree — the bootstrap page shows the handshake, and Three Routes lines resampling up against the Wald and Bayesian answers on the same data. Where the calculus runs out (medians, quantiles, awkward statistics), the loop carries on alone.