If sums of Normal Distributions are Normal :Why are the weighted sums of Normal Distributions non-Normal?

Question

I posted this question here Do 2 Normal Distributions always produce another Normal Distribution? where it was suggested to me that Characteristic Functions can be used to show that the distribution for the sums of independent Normal Distributions are still Normal.

I had originally tried to prove this using Convolutions, but I think this is easier to show using Characteristic Functions:

Define two random variables $X$ and $Y$ with probability distributions $f(x)$ and $f(y)$. Assuming that we can determine the Characteristic Functions for both of these Random Variables:

$$ \phi(t) = E[e^{itX}] $$

Then the sum of these two random variables $Z = X+Y$ has the following Characteristic Function

$$ \phi_Z(t) = \phi_X(t) \cdot \phi_Y(t) $$

Using this logic, it seems much easier to prove that the distribution for the sums of two Normal Distributions is still Normal:

$$ \text{Characteristic Function of a Normal Distribution:} \quad \phi(t) = e^{iμt - \frac{1}{2}σ^2t^2} $$

The Characteristic Functions of both Random Variables are:

$$ \phi_X(t) = e^{iμ1t - \frac{1}{2}σ1^2t^2} $$

$$ \phi_Y(t) = e^{iμ2t - \frac{1}{2}σ2^2t^2} $$

We can then use this to find out the Characteristic Function of the sum $Z$:

$$ \phi_Z(t) = \phi_X(t) \cdot \phi_Y(t) = e^{iμ1t - \frac{1}{2}σ1^2t^2} \cdot e^{iμ2t - \frac{1}{2}σ2^2t^2} $$

$$ \phi_Z(t) = e^{i(μ1+μ2)t - \frac{1}{2}(σ1^2+σ2^2)t^2} $$

Using the property that Characteristic Functions are unique, we can recognize that $Z$ is basically a Normal Distribution with $\mu = \mu_1 + \mu_2$ and $\sigma^2 = \sigma_1^2 + \sigma_2^2$.

This leads me to my actual question: Are the weighted sums of Normal Distributions also Normally Distributed?

I have read about Finite Mixture Models (e.g. https://en.wikipedia.org/wiki/Mixture_model) in which a probability distribution is defined as follows:

$$ f(x) = w1*\frac{1}{\sqrt{2\pi\sigma1^2}}e^{-\frac{(x-μ1)^2}{2\sigma1^2}} + w2*\frac{1}{\sqrt{2\pi\sigma2^2}}e^{-\frac{(x-μ2)^2}{2\sigma2^2}} + ... + wk*\frac{1}{\sqrt{2\pi\sigma_k^2}}e^{-\frac{(x-μ_k)^2}{2\sigma_k^2}} $$

$$ \sum_{i=1}^{n} w_i = 1 $$

$$ 0 \leq w_i \leq 1 \quad \text{for all } i $$

My initial thought was that if the sums of Normal Distributions are Normal, then shouldn't the Weighted Sums of Normal Distributions also be Normal? However on the other hand, I have seen visual representations of Finite Mixtures that don't look very Normal.

I tried to write the sum of Characteristic Functions (e.g. start with only two distributions), I got:

$$ \phi_{mixture}(t) = w1*\phi_{1}(t) + w2*\phi_{2}(t) $$

$$ \phi_{mixture}(t) = w1*e^{iμ1t - \frac{1}{2}σ^2t^2} + w2*e^{iμ2t - \frac{1}{2}σ^2t^2} $$

I am not sure if this result is Normal or Non-Normal. Can someone please help me out here? Is the above equation the Characteristic Function of a Normal Distribution?

Answer 1

The weighted sum of normal variables is also a normal variable. This can be shown without resorting to the characteristic function. For example consider two normal variables $X_1 \sim N(\mu_1, \sigma_1)$ and $X_2 \sim N(\mu_2, \sigma_2)$ and the weighted sum:

$Z = w_1 X_1 + w_2 X_2$

with $w_1$ and $w_2$ constants.

The variable $Y_1 = w_1 X_1$ follows a normal distribution, $Y_1 \sim N(w_1 \,\mu_1, w_1 \, \sigma_1)$. For the same reason, $Y_2 = w_2 \, X_2 \sim N(w_2 \, \mu_2, w_2 \, \sigma_2)$. Then $Z$, the unweighted sum of $Y_1$ and $Y_2$, is also normal. The mean of $Z$ is $\mu_Z = w_1 \,\mu_1 + w_2 \,\mu_2$, and the variance is $\sigma^2_Z = (w_1 \, \sigma_1)^2 + (w_2 \, \sigma_2)^2 + 2 \rho \, w_1 \, \sigma_1 \, w_2 \, \sigma_2$, where $\rho$ is the correlation coefficient between $X_1$ and $X_2$. If $X_1$ and $X_2$ are independent $\rho=0$, and $\sigma^2_Z = (w_1 \, \sigma_1)^2 + (w_2 \, \sigma_2)^2$.

The same reasoning can be applied inductively to the weighted sum of $n$ normal variables.

Answer 2

For general $f(x)$ as you mentioned before: $$f(x)=\sum_{i}w_if_i(x),$$ where $\sum_i w_i=1$ and $f_i(x)$ is a probability density function of normal distribution for all $i$, $f$ can be non-normal. This is called Gaussian Mixture Model (GMM).

$f$ could be Gaussian, if we simply choose only one $w_i$ to be $1$ and other $w_i$ to be $0$. $f$ is also Gaussian if all $f_i=f_0$, where $f_0$ is Gaussian.

Conversely, $f$ could be non-Gaussian. Consider $w_1=w_2=1/2$ and $f_1$ is the density of $\mathcal{N}(1,1^2)$ and $f_2$ is the density of $\mathcal{N}(-1,1^2)$. The density of $f$ is $$f=\frac{1}{2}f_1+\frac{1}{2}f_2=\frac{1}{\sqrt{2\pi}}(\frac{1}{2}e^{-\frac{(x+1)^2}{2}}+\frac{1}{2}e^{-\frac{(x-1)^2}{2}}).$$

$f$ is not a density of a Gaussian variable. This is because a density of Gaussian $\mathcal{N}(\mu,\sigma^2)$ only have one stationary point $x=\mu$ on $\mathbb{R}$, but $f$ has two stationary points. ($x_1=-1,x_2=1$)

Intuitively, the density of Gaussian has only one peak, but the mixture of Gaussian density has multiple peaks, which implies the mixture of Gaussian density is not Gaussian.

---

In general, if a random variable $X=\sum_{i}w_iX_i$, where $X_i$ are independent Gaussian random variables for all $i$, then $X$ is Gaussian, which can be proved by checking characteristic function of $X$.

In fact, we only need to show that (i) if $X$ is Gaussian, then $aX$ is Gaussian for scalar $a$ (ii) if $X_1,X_2$ are both Gaussian and independent, then $X_1+X_2$ is Gaussian

If (i) and (ii) both hold, then $X=\sum_{i}w_iX_i$ is Gaussian, since $X$ is nothing but the finite summations of scalar times Gaussian variables.

Proof of (i): Suppose $X\sim\mathcal{N}(\mu,\sigma^2)$. Then the characteristic function of $aX$ is $$\phi_{aX}(t)=\mathbb{E}(e^{itaX})=e^a\mathbb{E}(e^{itX})=e^a(e^{it\mu-\frac{1}{2}\sigma^2t^2})=e^{it(a\mu)-\frac{1}{2}a\sigma^2t^2},$$ which implies $aX\sim\mathcal{N}(a\mu,a\sigma^2)$.

Proof of (ii): In fact, (ii) holds if $X_1$ and $X_2$ form multivariate normal distribution (no need to be independent) and the general proof of this case can be referred to this post.

Suppose $X_i\sim\mathcal{N}(\mu_i,\sigma_i^2)$ for $i\in\{1,2\}$. The characteristic function of $X_1+X_2$ is $$\phi_{X_1+X_2}(t)=\mathbb{E}(e^{it(X_1+X_2)})=\mathbb{E}(e^{itX_1})\mathbb{E}(e^{itX_2})=e^{it\mu_1-\frac{1}{2}\sigma_1^2t^2}e^{it\mu_2-\frac{1}{2}\sigma_2^2t^2}=e^{it(\mu_1+\mu_2)-\frac{1}{2}(\sigma_1^2+\sigma_2^2)t^2},$$ which implies $X_1+X_2\sim\mathcal{N}(\mu_1+\mu_2,\sigma_1^2+\sigma_2^2)$.

Answer 3

Here is my attempt to answer my own question.

The characteristic function of a Gaussian random variable $X$ with mean $\mu$ and variance $\sigma^2$ is given by:

$$\phi_X(t) = e^{i\mu t - \frac{1}{2}\sigma^2t^2}$$

1. Weighted Sum of Gaussian Random Variables

If we have a weighted sum of Gaussian random variables $Y = a_1X_1 + a_2X_2 + ... + a_nX_n$, then the characteristic function of $Y$ is the product of the characteristic functions of the $X_i$:

$$\phi_Y(t) = \prod_{i=1}^{n} \phi_{X_i}(a_it) = \prod_{i=1}^{n} e^{ia_i\mu_i t - \frac{1}{2}a_i^2\sigma_i^2t^2}$$

$$\phi_Y(t) = e^{i(\sum_{i=1}^{n}a_i\mu_i)t - \frac{1}{2}(\sum_{i=1}^{n}a_i^2\sigma_i^2)t^2}$$

This looks like the Characteristic Function of a Gaussian

2. Weighted Sum of Gaussian Densities

For a weighted sum of Gaussian densities:

$$g(x) = a_1 f(x; \mu_1, \sigma_1^2) + a_2 f(x; \mu_2, \sigma_2^2)$$

The weighted sum of these characteristic functions:

$$\phi_g(t) = a_1\phi_{f_1}(t) + a_2\phi_{f_2}(t) = a_1e^{i\mu_1 t - \frac{1}{2}\sigma_1^2t^2} + a_2e^{i\mu_2 t - \frac{1}{2}\sigma_2^2t^2}$$

While this kind of resembles a Gaussian Characteristic Function, it's not exactly a Gaussian Characteristic Function. I think this is not a Gaussian characteristic function unless $\mu_1 = \mu_2$ and $\sigma_1^2 = \sigma_2^2$

While the math makes sense to me for now, I must admit that this is a paradox for me. How can the sums of Gaussian Random Variables necessarily always be Gaussian but the sums of Gaussian Densities not necessarily always be Gaussian?

I hope someone will clarify on this point ...

Answer 4

You're mixing up the distribution of the sum of random variables and the sum of the distribution of random variables. If $X$ and $Y$ are independent random variables with PDF $PDF_X, PDF_Y$ respectively, then the pdf of the former is $PDF_{X+Y}(t) = PDF_X \ast PDF_Y = \int_{\infty}^{\infty} PDF_X(u) PDF_Y(t-u)du$. Whereas the pdf of the latter is $PDF_X(t) + PDF_Y(t)$. That "the sum of normal distributions is normal" is a statement about the former meaning of sum; a Gaussian Mixture model is the latter.