# Relationship between PGFs and Moments?

I am trying to understand the relationship between Moments and Probability Generator Functions (PGF).

As I understand, the PGF $$G_X(s)$$ of a discrete random variable $$X$$ that takes non-negative integer values is defined as: $$G_X(s) = \mathbb{E}[s^X] = \sum_{k=0}^{\infty} P(X = k) s^k$$

I read this relationship between the kth Moment and PGFs:

$$G_X(s) = \mathbb{E}[s^X] = \sum_{k=0}^{\infty} P(X = k) s^k$$

• For k = 1: $$E[X] = G_X'(1)$$
• For k = 2: $$E[X^2] = G_X''(1) + G_X'(1)$$
• For k = 3: $$E[X^3] = G_X'''(1) + 3G_X''(1) + G_X'(1)$$
• etc.

I am trying to understand why this above relationship is true.

I tried to verify this relationship for the first and second moments:

1) First Moment (Mean)

Differentiate the PGF: $$G_X'(s) = \frac{d}{ds} G_X(s) = \frac{d}{ds} \left( \sum_{k=0}^{\infty} P(X = k) s^k \right)$$

Apply the derivative inside the summation: $$G_X'(s) = \sum_{k=0}^{\infty} P(X = k) \frac{d}{ds} (s^k)$$

Compute the derivative: $$\frac{d}{ds} (s^k) = k s^{k-1}$$

Substitute back into the summation: $$G_X'(s) = \sum_{k=0}^{\infty} P(X = k) k s^{k-1}$$

Evaluate at $$s = 1$$: $$G_X'(1) = \sum_{k=0}^{\infty} P(X = k) k \cdot 1^{k-1} = \sum_{k=0}^{\infty} k P(X = k)$$

This sum is the definition of the expected value $$\mathbb{E}[X]$$: $$G_X'(1) = \mathbb{E}[X]$$

2) Second Moment

Differentiate the PGF a second time $$G_X''(s) = \frac{d}{ds} G_X'(s) = \frac{d}{ds} \left( \sum_{k=0}^{\infty} P(X = k) k s^{k-1} \right)$$

Apply the derivative inside the summation: $$G_X''(s) = \sum_{k=0}^{\infty} P(X = k) k \frac{d}{ds} (s^{k-1})$$

Compute the derivative: $$\frac{d}{ds} (s^{k-1}) = (k-1) s^{k-2}$$

Substitute back into the summation: $$G_X''(s) = \sum_{k=0}^{\infty} P(X = k) k (k-1) s^{k-2}$$

Evaluate at $$s = 1$$: $$G_X''(1) = \sum_{k=0}^{\infty} P(X = k) k (k-1) \cdot 1^{k-2} = \sum_{k=0}^{\infty} k (k-1) P(X = k)$$

Doing some algebra, we can see that:

$$\mathbb{E}[X^2] = \sum_{k=0}^{\infty} k^2 P(X = k)$$

$$k^2 = k(k-1) + k$$

$$\sum_{k=0}^{\infty} k^2 P(X = k) = \sum_{k=0}^{\infty} \left[ k(k-1) + k \right] P(X = k)$$

$$\sum_{k=0}^{\infty} k^2 P(X = k) = \sum_{k=0}^{\infty} k (k-1) P(X = k) + \sum_{k=0}^{\infty} k P(X = k)$$

The first term, $$\sum_{k=0}^{\infty} k (k-1) P(X = k)$$, is exactly $$G_X''(1)$$.

The second term, $$\sum_{k=0}^{\infty} k P(X = k)$$, is the mean $$\mathbb{E}[X]$$, which we have from the first derivative evaluated at $$s = 1$$:

Thus in total, we can see that this is related to the first and second derivatives of the PGF:

$$\mathbb{E}[X^2] = G_X''(1) + G_X'(1)$$

1. Proof for the $$K^{th}$$ moment . I am not sure how to do this - I got stuck here is my attempt:

$$G_X(s) = \sum_{n=0}^{\infty} P(X = n) s^n$$

$$G_X'(s) = \sum_{n=0}^{\infty} P(X = n) n s^{n-1}$$

$$G_X''(s) = \sum_{n=0}^{\infty} P(X = n) n(n-1) s^{n-2}$$

We can see a pattern. Take the kth derivative:

$$G_X^{(k)}(s) = \sum_{n=k}^{\infty} P(X = n) n(n-1)(n-2)...(n-k+1) s^{n-k}$$

Evaluate this at s = 1:

$$G_X^{(k)}(1) = \sum_{n=k}^{\infty} P(X = n) n(n-1)(n-2)...(n-k+1)$$

Look at the kth moment of X:

$$E[X^k] = \sum_{n=0}^{\infty} n^k P(X = n)$$

I tried to expand $$n^k$$ using the falling factorial notation:

$$n^k = n(n-1)(n-2)...(n-k+1) + \text{lower order terms}$$

Substituting this into the expression for $$E[X^k]$$:

$$E[X^k] = \sum_{n=k}^{\infty} n(n-1)(n-2)...(n-k+1) P(X = n) + \sum_{n=0}^{\infty} (\text{lower order terms}) P(X = n)$$

The first term in this sum is exactly $$G_X^{(k)}(1)$$. The second term involves lower order derivatives of G_X evaluated at 1. So we can write:

$$E[X^k] = G_X^{(k)}(1) + \text{combination of lower order derivatives}$$

I am not sure if this proves the general relationship between the kth moment and the kth derivative of the PGF evaluated at $$s = 1$$.

As you've seen, taking the derivatives of the PGF gives you what are called the factorial moments

$$\mathbb{E}((X)_k) = G^{(k)}(1).$$

To get the moments from here requires expressing the polynomial $$x^n$$ in terms of the falling factorials $$(x)_n$$. This expansion has coefficients given by the Stirling numbers of the second kind

$$x^n = \sum_{k=0}^n S(n, k) (x)_k$$

which gives

$$\mathbb{E}(X^n) = \sum_{k=0}^n S(n, k) \mathbb{E}((X)_k) = \sum_{k=0}^n S(n, k) G^{(k)}(1).$$

An alternative way to prove this is to show that

$$\mathbb{E}(X^n) = \left( \left( s \frac{d}{ds} \right)^n G(s) \right) \bigg\vert_{s=1}$$

and figure out how to expand $$\left( s \frac{d}{ds} \right)^n$$ in terms of $$s^k \frac{d^k}{ds^k}$$. There's a combinatorial argument that gives the same answer as above, or one can write down a recursion.

• @ Qiaochu Yuan : Do you think what i have done this is correct? Commented Aug 1 at 5:39
• This is my other question on proving properties of PGFs: math.stackexchange.com/questions/4953130/… Commented Aug 1 at 5:40
• @wulasa: what you've done is correct so far but it's incomplete. You haven't said what the lower order terms are (and it's not so obvious; the Stirling numbers are not as famous as the binomial coefficients, eg). Commented Aug 1 at 5:41
• thank you ... can you please expand it if you have time? I am going to work on it as well... Commented Aug 1 at 5:42
• @Ryan: it means to apply the differential operator $s \frac{d}{ds}$ $n$ times. This operator has the property that $\left( s \frac{d}{ds} \right)^n s^m = m^n s^m$ so it is exactly the thing needed to get the moments. Commented Aug 1 at 7:10