# How to Define Higher-Order Terms Analogous to Expectation and Variance in Probability Theory?

Let $$X$$ be a random variable, with its expectation denoted as $$\mu^{(1)} := E[X]$$. Correspondingly, its variance is defined as $$\mu^{(2)} := \text{Var}(X) = E[(X - \mu^{(1)})^2]$$.

We have come up with an analogy that may or may not be appropriate: if we draw an analogy to the Taylor expansion, then the "expectation" corresponds to the "first-order term," and the "variance" corresponds to the "second-order term." Our question is: how should we define the "higher-order terms"?

Our proposed solution is to define \begin{align} \mu^{(i+1)} = E[(\cdots (((X - \mu^{(1)})^2-\mu^{(2)})^2 -\cdots)^2- \mu^{(i)})^2]. \end{align} Specifically, we observe that: \begin{align} E[((X - \mu^{(1)})^2 - \mu^{(2)})^2] = \text{Var}((X - \mu^{(1)})^2). \end{align} This definition appears to have some probabilistic significance. Therefore, our questions are:

1. Is this analogy appropriate, and does such a definition exist in probability theory?
2. If such a definition exists, could you provide some explanations and references?
3. If the above definition is not probabilistically meaningful, is there a meaningful higher-order definition (different from higher-order moments)?

• I think you're just thinking about the higher moments, defined like $$\mathbb{E}[(X-\mathbb{E}[X])^k].$$ For example, when $k=3$ and $4$, this is called the skew and kurtosis, respectively. Commented Jul 18 at 3:14
• @Alan: but that doesn't specialize to the expectation when $k = 1$. Commented Jul 18 at 3:44
• You might be looking for cumulants? Commented Jul 18 at 3:47

The higher-order generalizations of the expectation and variance are called the cumulants, $$\kappa_n(X)$$. They can be defined using the logarithm of the moment generating function:

$$K_X(t) = \log M_X(t) = \log \mathbb{E}(e^{tX}) = \sum_{n \ge 0} \kappa_n(X) \frac{t^n}{n!}.$$

So they are literally the terms in the Taylor series expansion of this logarithm. This can be used to write down an explicit formula for $$\kappa_n(X)$$ as a certain polynomial in the moments $$m_n(X) = \mathbb{E}(X^n)$$, generalizing the formula for the variance in terms of $$m_1$$ and $$m_2$$.

$$\kappa_1$$ is the expectation and $$\kappa_2, \kappa_3$$ are the second and third central moments, but $$\kappa_n, n \ge 4$$ is not a central moment; see Wikipedia.

One way to justify that these are the "correct" generalization of the expectation and the variance is that, like the expectation and variance, they are additive with respect to sums of independent random variables; that is, if $$X_1, \dots X_k$$ are independent we have

$$\kappa_n(X_1 + \dots + X_k) = \kappa_n(X_1) + \dots + \kappa_n(X_k).$$

This is not true of the higher moments or the higher central moments, and follows from taking the logarithm of the MGF identity $$M_{X_1 + \dots + X_k}(t) = M_{X_1}(t) \dots M_{X_k}(t)$$. I think there is even some result to the effect that $$\kappa_n$$ is (up to scale) the only homogeneous function of degree $$n$$ of the moments of a random variable with this property, but I don't know a proof off the top of my head so don't quote me on that.

Cumulants offers an interesting way of thinking about the central limit theorem. (The following is not a complete proof but it is suggestive.) If $$X$$ is a mean zero random variable with finite moments then take an infinite sequence $$X_1, X_2, \dots$$ of iid copies of it and consider the sequence of rescaled sums $$S_n = \frac{X_1 + \dots + X_n}{\sqrt{n}}$$, which have mean zero and the same variance as $$X$$. By the additivity property above, the cumulant generating function of $$S_n$$ is

\begin{align*} K_{S_n}(t) &= n K_X \left( \frac{t}{\sqrt{n}} \right)\\ &= \kappa_2(X) \frac{t^2}{2!} + n^{-\frac{1}{2}} \kappa_3(X) \frac{t^3}{3!} + n^{-1} \kappa_4(X) \frac{t^4}{4!} + \dots \end{align*}.

So we see that adding independent copies of $$X$$ and then rescaling so that the variance is fixed has the effect of scaling all the higher cumulants down. Naturally this means that as $$n \to \infty$$ we expect the higher cumulants to vanish - and this characterizes Gaussian distributions. I have been told that this is an example or maybe a toy model of renormalization group flow.