Relationship between PGFs and Moments?

Question

I am trying to understand the relationship between Moments and Probability Generator Functions (PGF).

As I understand, the PGF $ G_X(s)$ of a discrete random variable $ X$ that takes non-negative integer values is defined as: $$ G_X(s) = \mathbb{E}[s^X] = \sum_{k=0}^{\infty} P(X = k) s^k $$

I read this relationship between the kth Moment and PGFs:

$$G_X(s) = \mathbb{E}[s^X] = \sum_{k=0}^{\infty} P(X = k) s^k $$

• For k = 1: $ E[X] = G_X'(1) $
• For k = 2: $ E[X^2] = G_X''(1) + G_X'(1) $
• For k = 3: $ E[X^3] = G_X'''(1) + 3G_X''(1) + G_X'(1) $
• etc.

I am trying to understand why this above relationship is true.

I tried to verify this relationship for the first and second moments:

1) First Moment (Mean)

Differentiate the PGF: $$ G_X'(s) = \frac{d}{ds} G_X(s) = \frac{d}{ds} \left( \sum_{k=0}^{\infty} P(X = k) s^k \right) $$

Apply the derivative inside the summation: $$ G_X'(s) = \sum_{k=0}^{\infty} P(X = k) \frac{d}{ds} (s^k) $$

Compute the derivative: $$ \frac{d}{ds} (s^k) = k s^{k-1} $$

Substitute back into the summation: $$ G_X'(s) = \sum_{k=0}^{\infty} P(X = k) k s^{k-1} $$

Evaluate at $ s = 1$: $$ G_X'(1) = \sum_{k=0}^{\infty} P(X = k) k \cdot 1^{k-1} = \sum_{k=0}^{\infty} k P(X = k) $$

This sum is the definition of the expected value $ \mathbb{E}[X]$: $$ G_X'(1) = \mathbb{E}[X] $$

2) Second Moment

Differentiate the PGF a second time $$ G_X''(s) = \frac{d}{ds} G_X'(s) = \frac{d}{ds} \left( \sum_{k=0}^{\infty} P(X = k) k s^{k-1} \right) $$

Apply the derivative inside the summation: $$ G_X''(s) = \sum_{k=0}^{\infty} P(X = k) k \frac{d}{ds} (s^{k-1}) $$

Compute the derivative: $$ \frac{d}{ds} (s^{k-1}) = (k-1) s^{k-2} $$

Substitute back into the summation: $$ G_X''(s) = \sum_{k=0}^{\infty} P(X = k) k (k-1) s^{k-2} $$

Evaluate at $ s = 1$: $$ G_X''(1) = \sum_{k=0}^{\infty} P(X = k) k (k-1) \cdot 1^{k-2} = \sum_{k=0}^{\infty} k (k-1) P(X = k) $$

Doing some algebra, we can see that:

$$ \mathbb{E}[X^2] = \sum_{k=0}^{\infty} k^2 P(X = k) $$

$$ k^2 = k(k-1) + k $$

$$ \sum_{k=0}^{\infty} k^2 P(X = k) = \sum_{k=0}^{\infty} \left[ k(k-1) + k \right] P(X = k) $$

$$ \sum_{k=0}^{\infty} k^2 P(X = k) = \sum_{k=0}^{\infty} k (k-1) P(X = k) + \sum_{k=0}^{\infty} k P(X = k) $$

The first term, $ \sum_{k=0}^{\infty} k (k-1) P(X = k) $, is exactly $ G_X''(1) $.

The second term, $ \sum_{k=0}^{\infty} k P(X = k) $, is the mean $ \mathbb{E}[X] $, which we have from the first derivative evaluated at $ s = 1 $:

Thus in total, we can see that this is related to the first and second derivatives of the PGF:

$$ \mathbb{E}[X^2] = G_X''(1) + G_X'(1) $$

3. Proof for the $K^{th}$ moment . I am not sure how to do this - I got stuck here is my attempt:

$$ G_X(s) = \sum_{n=0}^{\infty} P(X = n) s^n $$

$$ G_X'(s) = \sum_{n=0}^{\infty} P(X = n) n s^{n-1} $$

$$ G_X''(s) = \sum_{n=0}^{\infty} P(X = n) n(n-1) s^{n-2} $$

We can see a pattern. Take the kth derivative:

$$ G_X^{(k)}(s) = \sum_{n=k}^{\infty} P(X = n) n(n-1)(n-2)...(n-k+1) s^{n-k} $$

Evaluate this at s = 1:

$$ G_X^{(k)}(1) = \sum_{n=k}^{\infty} P(X = n) n(n-1)(n-2)...(n-k+1) $$

Look at the kth moment of X:

$$ E[X^k] = \sum_{n=0}^{\infty} n^k P(X = n) $$

I tried to expand $n^k$ using the falling factorial notation:

$$ n^k = n(n-1)(n-2)...(n-k+1) + \text{lower order terms} $$

Substituting this into the expression for $E[X^k]$:

$$ E[X^k] = \sum_{n=k}^{\infty} n(n-1)(n-2)...(n-k+1) P(X = n) + \sum_{n=0}^{\infty} (\text{lower order terms}) P(X = n) $$

The first term in this sum is exactly $G_X^{(k)}(1)$. The second term involves lower order derivatives of G_X evaluated at 1. So we can write:

$$ E[X^k] = G_X^{(k)}(1) + \text{combination of lower order derivatives} $$

I am not sure if this proves the general relationship between the kth moment and the kth derivative of the PGF evaluated at $s = 1$.

Can someone please help me correct my proof?

Answer 1

As you've seen, taking the derivatives of the PGF gives you what are called the factorial moments

$$\mathbb{E}((X)_k) = G^{(k)}(1).$$

To get the moments from here requires expressing the polynomial $x^n$ in terms of the falling factorials $(x)_n$. This expansion has coefficients given by the Stirling numbers of the second kind

$$x^n = \sum_{k=0}^n S(n, k) (x)_k$$

which gives

$$\mathbb{E}(X^n) = \sum_{k=0}^n S(n, k) \mathbb{E}((X)_k) = \sum_{k=0}^n S(n, k) G^{(k)}(1).$$

An alternative way to prove this is to show that

$$\mathbb{E}(X^n) = \left( \left( s \frac{d}{ds} \right)^n G(s) \right) \bigg\vert_{s=1}$$

and figure out how to expand $\left( s \frac{d}{ds} \right)^n$ in terms of $s^k \frac{d^k}{ds^k}$. There's a combinatorial argument that gives the same answer as above, or one can write down a recursion.