I have two multivariate Gaussians each defined by mean vectors and Covariance matrices (diagonal matrices). I want to merge them to have a single Gaussian i.e. I assume there is only one Gaussian but I separated observations randomly into two groups to get two different Gaussians which are not too different than each other.

Since I know the number of observations in each of two Gaussians, combined mean estimation is straight forward : $\frac{n_1\mu_1 + n_2\mu_2}{n_1+n_2}$

But, what about the Covariance matrix?

Thanks

EDIT:

The question was confusing in the original post, especially the "merging Gaussians" part. Maybe the following paragraph would be a better choice.

I have two sets of observations drawn from two multivariate Gaussians each defined by mean vectors and Covariance matrices (diagonal matrices). I want to merge the observations to have a single sample, and I assume to have another Gaussian (i.e. I assume initially there was only a single Gaussian, and observations were separated into two groups to get two different Gaussians).

  • Ok I solved it :) Since covariance matrix is diagonal we can assume having multiple univariates. And then variance combination is as mu = (n1*mu1 + n2*mu2) / (n1+n2) sigma^2 = (((sigma1^2 + mu1^2)*n1 + (sigma2^2 + mu2^2)*n2) / (n1+n2)) - mu^2 ps: I used the equation sigma^2 = E[x^2] - E[x]^2 thanks again – ahmethungari Jul 26 '13 at 21:59
  • You could post this as an answer (preferably formatted in $\LaTeX$) and accept it. This is encouraged by this discussion on meta – Ross Millikan Jul 26 '13 at 22:04
  • Unfortunately, it did not let me to do so since I do not have enough reputation. I was required to wait some time. – ahmethungari Jul 28 '13 at 1:04
up vote 5 down vote accepted

Ok I solved it :)

Since covariance matrix is diagonal we can assume having multiple univariates. And then variance combination is as

$$\hat{\mu} = \frac{n_1\mu_1 + n_2\mu_2}{n_1+n_2}$$

$$\hat{\sigma}^2 = \frac{(\sigma_1^2 + \mu_1^2)n_1 + (\sigma_2^2 + \mu_2^2)n_2}{ (n_1+n_2)} - \hat{\mu}^2$$

Here, I used $\sigma^2 = E[x^2] - E[x]^2$

thanks again

I might be wrong or misinterpreted the question, but trying to reproduce the result in the accepted and upvoted answer, I get a different result:

Let $x \sim N(\mu, \sigma^2)$. From the definition of the variance follows $$ \sigma^2 = E[x^2] - E[x]^2 = E[x^2] - \mu^2 $$$$ or \quad E[x^2] = \sigma^2 + \mu^2 $$

Now let $x$ be the random variable defined as the weighted average $x = \frac{n_1x_1 + n_2x_2}{n_1 + n_2}$, where $x_1 \sim N(\mu_1, \sigma_1^2)$ and $x_2 \sim N(\mu_2, \sigma_2^2)$ are independent.

We have easily $$ E[x] = \frac{n_1\mu_1 + n_2\mu_2}{n_1 + n_2} := \mu $$

By the above formula for $E[x_1^2]$ and $E[x_2^2]$, and since $x_1$ and $x_2$ are independent ($E[x_1x_2] = E[x_1]E[x_2]$) we have

\begin{align} E[x^2] &= E[(\frac{n_1x_1 + n_2x_2}{n_1 + n_2})^2] \\ &= \frac{1}{(n_1 + n_2)^2} E[n_1^2 x_1^2 + n_2^2 x_2^2 + 2n_1n_2x_1x_2] \\ &= \frac{1}{(n_1 + n_2)^2} (n_1^2 E[x_1^2] + n_2^2 E[x_2^2] + 2n_1n_2E[x_1]E[x_2]) \\ &= \frac{(\sigma_1^2 + \mu_1^2)n_1^2 + (\sigma_2^2 + \mu_2^2)n_2^2 + 2n_1n_2\mu_1\mu_2}{(n_1+n_2)^2} \\ &= \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} \end{align}

We can use this to calculate the pooled variance: $$ \sigma^2 = E[x^2] - E[x]^2 = \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} - \mu $$


In the multivariate case, if the covariance matrix is diagonal, we can apply this formula on each dimension separately. Otherwise, let $X \sim N(\mu, \Sigma)$, where $\mu \in \mathbb{R}^n$ and $\Sigma \in \mathbb{R}^{n \times n}$.

Again start with the general definition of the covariance matrix that gives

$$ \Sigma = E[(X-\mu)(X-\mu)^T] = E[XX^T] - \mu \mu^T $$$$ or \quad E[XX^T] = \Sigma + \mu\mu^T $$

Let $X$ be the random variable defined as the weighted average $X = \frac{n_1X_1 + n_2X_2}{n_1 + n_2}$, where $X_1 \sim N(\mu_1, \Sigma_1)$ and $X_2 \sim N(\mu_2, \Sigma_2)$ are independent.

To simplify, let $D = \frac{1}{(n_1 + n_2)^2}$ . We have now

\begin{align*} E[XX^T] &= D \cdot E[(n_1X_1 + n_2X_2)(n_1X_1 + n_2X_2)^T] \\ &= D \cdot E[n_1^2 X_1X_1^T + n_2^2 X_2X_2^T + 2n_1n_2X_1X_2^T] \\ &= D \cdot (n_1^2 E[X_1X_1^T] + n_2^2 E[X_2X_2^T] + 2n_1n_2E[X_1]E[X_2^T]) \\ &= D \cdot (n_1^2 (\Sigma_1 + \mu_1\mu_1^T) + n_2^2 (\Sigma_2 + \mu_2\mu_2^T) + 2n_1n_2\mu_1\mu_2^T) \\ &= D \cdot (n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T) \end{align*}

Finally the pooled covariance matrix is: $$ \Sigma = E[XX^T] - E[X]E[X]^T = \frac{n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T}{(n_1 + n_2)^2} - \mu\mu^T $$

  • Welcome to MSE. Nice first post! – José Carlos Santos Jul 20 '17 at 10:53
  • Sorry for not much clear question. With merge, I had simply meant that we concatenate (put together) two sets of observations (with sizes $n_1$ and $n_1$) and recalculate mean and variance. So there is no assumption of defining a new variable as the weighted average of the older two. Hence, you simply take the average of $E[x_1^2]$ and $E[x_2^2]$ when calculating new $E[x^2]$. Again, sorry for confusion. – ahmethungari Aug 17 '17 at 0:20
  • @ahmethungari I disagree: you give a formula for E[x^2] - E[x]^2 but you never say what "x" is, and it is not obvious. In particular, with your result, x cannot be the intuitive weighted average I used above. – JulienD Aug 18 '17 at 7:38

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.