Tutorial 4: Handling Overdispersion

Step 3: Choose the Distribution Family

✓

Remember: Our Data is Overdispersed!

From Tutorial 3, we found Deviance/df = 524.1 (should be ~1). This means the variance in our data is 524 times larger than Poisson assumes! The consequence: Poisson gives standard errors that are far too small.

Step 3: Select the Distribution Family

Which distribution properly models overdispersed count data?

Hint: We need a distribution where Variance > Mean is allowed.

Negative Binomial

Y ~ NegBin(μ, θ)

Variance: μ + μ²/θ

Counts with extra dispersion parameter θ. Allows Var > Mean.

Poisson

Y ~ Poisson(μ)

Variance: μ (equals mean)

Standard count distribution. Assumes Variance = Mean.

Quasi-Poisson

Y ~ QuasiPoisson(μ, φ)

Variance: φ × μ

Quick fix - inflates SEs by φ. Not a proper probability model.

Gaussian

Y ~ Normal(μ, σ²)

Variance: σ² (constant)

For continuous outcomes. Allows negative values.

✓ Negative Binomial Selected

The Negative Binomial distribution properly handles overdispersed count data by estimating a dispersion parameter θ from the data.

← Back to Link Function

✔ Correct! Negative Binomial handles overdispersion.

The Negative Binomial distribution is perfect for overdispersed count data:

Variance = μ + μ²/θ: Always larger than the mean
Dispersion parameter θ: Estimated from data, controls extra variance
Proper probability model: Unlike quasi-Poisson, has a full likelihood
Reduces to Poisson: When θ → ∞, NegBin → Poisson

Metric	Poisson (Tutorial 3)	Negative Binomial
Deviance/df	524.1	1.04
AIC	387,415	12,840
SE(temp)	0.0032	0.0813

        Key insight: The standard errors are ~25x larger with Negative Binomial.
        This gives honest uncertainty estimates - Poisson was massively overconfident!
      

❌ We already know Poisson doesn't fit!

We tried Poisson in Tutorial 3 and discovered severe problems:

Deviance/df = 524: Should be ~1 for a good fit
Variance = Mean assumption: Our data violates this drastically
Underestimated SEs: p-values are unreliable (too optimistic)

        The overdispersion problem: Poisson assumes Var(Y) = μ.
        But our bike rental counts have far more variability than that.
        Days with similar predicted counts can have wildly different actual counts.
        
        Try selecting Negative Binomial to properly model this extra variance.

⚠ Quick fix, but not ideal.

The Quasi-Poisson is a pragmatic workaround:

Inflates SEs: Multiplies by √φ to account for overdispersion
Same coefficients: Point estimates identical to Poisson
No proper likelihood: Can't compute AIC or do likelihood ratio tests

        When to use Quasi-Poisson:
        Quick exploratory analysis
When you just need corrected SEs
Teaching/simple demonstrations

        Why Negative Binomial is better:
        Proper probability model with full likelihood
Can compute AIC, do model selection
More flexible variance structure: Var = μ + μ²/θ

        Try selecting Negative Binomial for the proper solution.
      

❌ Wrong data type - counts, not continuous!

The Gaussian distribution is wrong for several reasons:

Allows negative values: You can't have -100 bike rentals
Continuous, not discrete: Counts are integers (0, 1, 2, ...)
Constant variance: Assumes same spread at all levels of μ

        When to use Gaussian: Tutorial 1 (heart rate) was continuous and
        could theoretically take any value. Bike rental counts are fundamentally different.
        
        Try selecting Negative Binomial - it's designed for overdispersed counts.