STA 310 - Spring 2024 - Unifying theory of GLMs

Many models; one family

We have studied models for a variety of response variables

Least squares (Normal)
Logistic (Bernoulli, Binomial, Multinomial)
Log-linear (Poisson, Negative Binomial)

These models are all examples of generalized linear models.

GLMs have a similar structure for their likelihoods, MLEs, variances, so we can use a generalized approach to find the model estimates and associated uncertainty.

Components of a GLM

Nelder and Wedderburn (1972) defines a broad class of models called generalized linear models that generalizes multiple linear regression. GLMs are characterized by three components:

1️⃣ Response variable with parameter $θ$ whose probability function can be written in exponential family form (random component)

2️⃣ A linear combination of predictors, $η = β_{1} x_{1} + β_{2} x_{2} + \dots + β_{p} x_{p}$ (systematic component)

3️⃣ A link function $g (θ)$ that connects $θ$ to $η$

One-parameter exponential family form

Suppose a probability (mass or density) function has a parameter $θ$ . It is said to have a one-parameter exponential family form if

✅ The support (set of possible values) does not depend on $θ$ , and

✅ The probability function can be written in the following form

$f (y; θ) = e^{[a (y) b (θ) + c (θ) + d (y)]}$

Mean and variance

On-parameter exponential family form

$f (y; θ) = e^{[a (y) b (θ) + c (θ) + d (y)]}$

Using this form:

$E (Y) = - \frac{c^{'} (θ)}{b^{'} (θ)} V a r (Y) = \frac{b^{″} (θ) c^{'} (θ) - c^{″} (θ) b^{'} (θ)}{[b^{'} (θ)]^{3}}$

Poisson in one-parameter exponential family form

$P (Y = y) = \frac{e^{- λ} λ^{y}}{y!} y = 0, 1, 2, \dots, \infty$

$\begin{aligned} P (Y = y) & = e^{- λ} e^{y \log (λ)} e^{- \log (y!)} \\ = e^{y \log (λ) - λ - \log (y!)} \end{aligned}$

Recall the form: $f (y; θ) = e^{[a (y) b (θ) + c (θ) + d (y)]}$ , where the parameter $θ = λ$ for the Poisson distribution

$a (y) = y$
$b (λ) = \log (λ)$
$c (λ) = - λ$
$d (y) = - \log (y!)$

Poisson in exponential family form

✅ The support for the Poisson distribution is $y = 0, 1, 2, \dots, \infty$ . This does not depend on the parameter $λ$ .

✅ The probability mass function can be written in the form $f (y; θ) = e^{[a (y) b (θ) + c (θ) + d (y)]}$

The Poisson distribution can be written in one-parameter exponential family form.

Canonical link

Suppose there is a response variable $Y$ from a distribution with parameter $θ$ and a set of predictors that can be written as a linear combination $η = β_{0} + \sum_{j = 1}^{p} β_{j} x_{j} = β_{0} + β_{1} x_{1} + β_{2} x_{2} + \dots + β_{p} x_{p}$

A link function, $g ()$ , is a monotonic and differentiable function that connects $θ$ to $η$
When working with a member of the one-parameter exponential family, $b (θ)$ is called the canonical link
- Most commonly used link function

Canonical link for Poisson

Recall the exponential family form:

$P (Y = y) = e^{y \log (λ) - λ - \log (y!)}$

then the canonical link is $b (λ) = \log (λ)$

GLM framework: Poisson response variable

1️⃣ Response variable with parameter $θ$ whose probability function can be written in exponential family form

$P (Y = y) = e^{y \log (λ) - λ - \log (y!)}$

2️⃣ A linear combination of predictors, $η = β_{0} + β_{1} x_{1} + β_{2} x_{2} + \dots + β_{p} x_{p}$

3️⃣ A function $g (λ)$ that connects $λ$ and $η$

$\log (λ) = η = β_{0} + β_{1} x_{1} + β_{2} x_{2} + \dots + β_{p} x_{p}$

Activity: Generalized linear models

For your group’s distribution

Write the pmf or pdf in one-parameter exponential form.
Describe an example of a setting where this random variable may be used.
Identify the canonical link function.

Activity: Generalized linear models

Distributions

Exponential
Gamma (with fixed $r$ )
Geometric
Binary

See BMLR - Section 3.6 for details on the distributions.

If your group finishes early, try completing the exercise for another distribution.

08:00

Using the exponential family form

The one-parameter exponential family form is utilized for

Calculating MLEs of coefficients (recall iteratively reweighted least squares)
Inference for coefficients
Likelihood ratio and drop-in-deviance tests

The specific calculations are beyond the scope of this course. See Section 4.6 of Dunn, Smyth, et al. (2018) for more detail (available at Duke library).

Unifying theory of GLMs

Announcements

Topics

Unifying theory of GLMs

Many models; one family

Components of a GLM

One-parameter exponential family form

Mean and variance

Poisson in one-parameter exponential family form

Poisson in exponential family form

Canonical link

Canonical link for Poisson

GLM framework: Poisson response variable

Activity: Generalized linear models

Activity: Generalized linear models

Using the exponential family form

References