# How to merge two Gaussians

I have two multivariate Gaussians each defined by mean vectors and Covariance matrices (diagonal matrices). I want to merge them to have a single Gaussian i.e. I assume there is only one Gaussian but I separated observations randomly into two groups to get two different Gaussians which are not too different than each other.

Since I know the number of observations in each of two Gaussians, combined mean estimation is straight forward : $\frac{n_1\mu_1 + n_2\mu_2}{n_1+n_2}$

But, what about the Covariance matrix?

Thanks

EDIT:

The question was confusing in the original post, especially the "merging Gaussians" part. Maybe the following paragraph would be a better choice.

I have two sets of observations drawn from two multivariate Gaussians each defined by mean vectors and Covariance matrices (diagonal matrices). I want to merge the observations to have a single sample, and I assume to have another Gaussian (i.e. I assume initially there was only a single Gaussian, and observations were separated into two groups to get two different Gaussians).

• Ok I solved it :) Since covariance matrix is diagonal we can assume having multiple univariates. And then variance combination is as mu = (n1*mu1 + n2*mu2) / (n1+n2) sigma^2 = (((sigma1^2 + mu1^2)*n1 + (sigma2^2 + mu2^2)*n2) / (n1+n2)) - mu^2 ps: I used the equation sigma^2 = E[x^2] - E[x]^2 thanks again – ahmethungari Jul 26 '13 at 21:59
• You could post this as an answer (preferably formatted in $\LaTeX$) and accept it. This is encouraged by this discussion on meta – Ross Millikan Jul 26 '13 at 22:04
• Unfortunately, it did not let me to do so since I do not have enough reputation. I was required to wait some time. – ahmethungari Jul 28 '13 at 1:04

Ok I solved it :)

Since covariance matrix is diagonal we can assume having multiple univariates. And then variance combination is as

$$\hat{\mu} = \frac{n_1\mu_1 + n_2\mu_2}{n_1+n_2}$$

$$\hat{\sigma}^2 = \frac{(\sigma_1^2 + \mu_1^2)n_1 + (\sigma_2^2 + \mu_2^2)n_2}{ (n_1+n_2)} - \hat{\mu}^2$$

Here, I used $\sigma^2 = E[x^2] - E[x]^2$

thanks again

I might be wrong or misinterpreted the question, but trying to reproduce the result in the accepted and upvoted answer, I get a different result:

Let $x \sim N(\mu, \sigma^2)$. From the definition of the variance follows $$\sigma^2 = E[x^2] - E[x]^2 = E[x^2] - \mu^2$$$$or \quad E[x^2] = \sigma^2 + \mu^2$$

Now let $x$ be the random variable defined as the weighted average $x = \frac{n_1x_1 + n_2x_2}{n_1 + n_2}$, where $x_1 \sim N(\mu_1, \sigma_1^2)$ and $x_2 \sim N(\mu_2, \sigma_2^2)$ are independent.

We have easily $$E[x] = \frac{n_1\mu_1 + n_2\mu_2}{n_1 + n_2} := \mu$$

By the above formula for $E[x_1^2]$ and $E[x_2^2]$, and since $x_1$ and $x_2$ are independent ($E[x_1x_2] = E[x_1]E[x_2]$) we have

\begin{align} E[x^2] &= E[(\frac{n_1x_1 + n_2x_2}{n_1 + n_2})^2] \\ &= \frac{1}{(n_1 + n_2)^2} E[n_1^2 x_1^2 + n_2^2 x_2^2 + 2n_1n_2x_1x_2] \\ &= \frac{1}{(n_1 + n_2)^2} (n_1^2 E[x_1^2] + n_2^2 E[x_2^2] + 2n_1n_2E[x_1]E[x_2]) \\ &= \frac{(\sigma_1^2 + \mu_1^2)n_1^2 + (\sigma_2^2 + \mu_2^2)n_2^2 + 2n_1n_2\mu_1\mu_2}{(n_1+n_2)^2} \\ &= \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} \end{align}

We can use this to calculate the pooled variance: $$\sigma^2 = E[x^2] - E[x]^2 = \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} - \mu$$

In the multivariate case, if the covariance matrix is diagonal, we can apply this formula on each dimension separately. Otherwise, let $X \sim N(\mu, \Sigma)$, where $\mu \in \mathbb{R}^n$ and $\Sigma \in \mathbb{R}^{n \times n}$.

Again start with the general definition of the covariance matrix that gives

$$\Sigma = E[(X-\mu)(X-\mu)^T] = E[XX^T] - \mu \mu^T$$$$or \quad E[XX^T] = \Sigma + \mu\mu^T$$

Let $X$ be the random variable defined as the weighted average $X = \frac{n_1X_1 + n_2X_2}{n_1 + n_2}$, where $X_1 \sim N(\mu_1, \Sigma_1)$ and $X_2 \sim N(\mu_2, \Sigma_2)$ are independent.

To simplify, let $D = \frac{1}{(n_1 + n_2)^2}$ . We have now

\begin{align*} E[XX^T] &= D \cdot E[(n_1X_1 + n_2X_2)(n_1X_1 + n_2X_2)^T] \\ &= D \cdot E[n_1^2 X_1X_1^T + n_2^2 X_2X_2^T + 2n_1n_2X_1X_2^T] \\ &= D \cdot (n_1^2 E[X_1X_1^T] + n_2^2 E[X_2X_2^T] + 2n_1n_2E[X_1]E[X_2^T]) \\ &= D \cdot (n_1^2 (\Sigma_1 + \mu_1\mu_1^T) + n_2^2 (\Sigma_2 + \mu_2\mu_2^T) + 2n_1n_2\mu_1\mu_2^T) \\ &= D \cdot (n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T) \end{align*}

Finally the pooled covariance matrix is: $$\Sigma = E[XX^T] - E[X]E[X]^T = \frac{n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T}{(n_1 + n_2)^2} - \mu\mu^T$$

• Welcome to MSE. Nice first post! – José Carlos Santos Jul 20 '17 at 10:53
• Sorry for not much clear question. With merge, I had simply meant that we concatenate (put together) two sets of observations (with sizes $n_1$ and $n_1$) and recalculate mean and variance. So there is no assumption of defining a new variable as the weighted average of the older two. Hence, you simply take the average of $E[x_1^2]$ and $E[x_2^2]$ when calculating new $E[x^2]$. Again, sorry for confusion. – ahmethungari Aug 17 '17 at 0:20
• @ahmethungari I disagree: you give a formula for E[x^2] - E[x]^2 but you never say what "x" is, and it is not obvious. In particular, with your result, x cannot be the intuitive weighted average I used above. – JulienD Aug 18 '17 at 7:38