9
$\begingroup$

I have two multivariate Gaussians each defined by mean vectors and Covariance matrices (diagonal matrices). I want to merge them to have a single Gaussian i.e. I assume there is only one Gaussian but I separated observations randomly into two groups to get two different Gaussians which are not too different than each other.

Since I know the number of observations in each of two Gaussians, combined mean estimation is straight forward : $\frac{n_1\mu_1 + n_2\mu_2}{n_1+n_2}$

But, what about the Covariance matrix?

Thanks

EDIT:

The question was confusing in the original post, especially the "merging Gaussians" part. Maybe the following paragraph would be a better choice.

I have two sets of observations drawn from two multivariate Gaussians each defined by mean vectors and Covariance matrices (diagonal matrices). I want to merge the observations to have a single sample, and I assume to have another Gaussian (i.e. I assume initially there was only a single Gaussian, and observations were separated into two groups to get two different Gaussians).

$\endgroup$
3
  • $\begingroup$ Ok I solved it :) Since covariance matrix is diagonal we can assume having multiple univariates. And then variance combination is as mu = (n1*mu1 + n2*mu2) / (n1+n2) sigma^2 = (((sigma1^2 + mu1^2)*n1 + (sigma2^2 + mu2^2)*n2) / (n1+n2)) - mu^2 ps: I used the equation sigma^2 = E[x^2] - E[x]^2 thanks again $\endgroup$ Jul 26, 2013 at 21:59
  • $\begingroup$ You could post this as an answer (preferably formatted in $\LaTeX$) and accept it. This is encouraged by this discussion on meta $\endgroup$ Jul 26, 2013 at 22:04
  • $\begingroup$ Unfortunately, it did not let me to do so since I do not have enough reputation. I was required to wait some time. $\endgroup$ Jul 28, 2013 at 1:04

4 Answers 4

10
$\begingroup$

Ok I solved it :)

Since covariance matrix is diagonal we can assume having multiple univariates. And then variance combination is as

$$\hat{\mu} = \frac{n_1\mu_1 + n_2\mu_2}{n_1+n_2}$$

$$\hat{\sigma}^2 = \frac{(\sigma_1^2 + \mu_1^2)n_1 + (\sigma_2^2 + \mu_2^2)n_2}{ (n_1+n_2)} - \hat{\mu}^2$$

Here, I used $\sigma^2 = E[x^2] - E[x]^2$

thanks again

$\endgroup$
1
4
$\begingroup$

I might be wrong or misinterpreted the question, but trying to reproduce the result in the accepted and upvoted answer, I get a different result:

Let $x \sim N(\mu, \sigma^2)$. From the definition of the variance follows $$ \sigma^2 = E[x^2] - E[x]^2 = E[x^2] - \mu^2 $$$$ or \quad E[x^2] = \sigma^2 + \mu^2 $$

Now let $x$ be the random variable defined as the weighted average $x = \frac{n_1x_1 + n_2x_2}{n_1 + n_2}$, where $x_1 \sim N(\mu_1, \sigma_1^2)$ and $x_2 \sim N(\mu_2, \sigma_2^2)$ are independent.

We have easily $$ E[x] = \frac{n_1\mu_1 + n_2\mu_2}{n_1 + n_2} := \mu $$

By the above formula for $E[x_1^2]$ and $E[x_2^2]$, and since $x_1$ and $x_2$ are independent ($E[x_1x_2] = E[x_1]E[x_2]$) we have

\begin{align} E[x^2] &= E[(\frac{n_1x_1 + n_2x_2}{n_1 + n_2})^2] \\ &= \frac{1}{(n_1 + n_2)^2} E[n_1^2 x_1^2 + n_2^2 x_2^2 + 2n_1n_2x_1x_2] \\ &= \frac{1}{(n_1 + n_2)^2} (n_1^2 E[x_1^2] + n_2^2 E[x_2^2] + 2n_1n_2E[x_1]E[x_2]) \\ &= \frac{(\sigma_1^2 + \mu_1^2)n_1^2 + (\sigma_2^2 + \mu_2^2)n_2^2 + 2n_1n_2\mu_1\mu_2}{(n_1+n_2)^2} \\ &= \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} \end{align}

We can use this to calculate the pooled variance: $$ \sigma^2 = E[x^2] - E[x]^2 = \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} - \mu^2 $$


In the multivariate case, if the covariance matrix is diagonal, we can apply this formula on each dimension separately. Otherwise, let $X \sim N(\mu, \Sigma)$, where $\mu \in \mathbb{R}^n$ and $\Sigma \in \mathbb{R}^{n \times n}$.

Again start with the general definition of the covariance matrix that gives

$$ \Sigma = E[(X-\mu)(X-\mu)^T] = E[XX^T] - \mu \mu^T $$$$ or \quad E[XX^T] = \Sigma + \mu\mu^T $$

Let $X$ be the random variable defined as the weighted average $X = \frac{n_1X_1 + n_2X_2}{n_1 + n_2}$, where $X_1 \sim N(\mu_1, \Sigma_1)$ and $X_2 \sim N(\mu_2, \Sigma_2)$ are independent.

To simplify, let $D = \frac{1}{(n_1 + n_2)^2}$ . We have now

\begin{align*} E[XX^T] &= D \cdot E[(n_1X_1 + n_2X_2)(n_1X_1 + n_2X_2)^T] \\ &= D \cdot E[n_1^2 X_1X_1^T + n_2^2 X_2X_2^T + 2n_1n_2X_1X_2^T] \\ &= D \cdot (n_1^2 E[X_1X_1^T] + n_2^2 E[X_2X_2^T] + 2n_1n_2E[X_1]E[X_2^T]) \\ &= D \cdot (n_1^2 (\Sigma_1 + \mu_1\mu_1^T) + n_2^2 (\Sigma_2 + \mu_2\mu_2^T) + 2n_1n_2\mu_1\mu_2^T) \\ &= D \cdot (n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T) \end{align*}

Finally the pooled covariance matrix is: $$ \Sigma = E[XX^T] - E[X]E[X]^T = \frac{n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T}{(n_1 + n_2)^2} - \mu\mu^T $$

$\endgroup$
6
  • $\begingroup$ Welcome to MSE. Nice first post! $\endgroup$ Jul 20, 2017 at 10:53
  • $\begingroup$ Sorry for not much clear question. With merge, I had simply meant that we concatenate (put together) two sets of observations (with sizes $n_1$ and $n_1$) and recalculate mean and variance. So there is no assumption of defining a new variable as the weighted average of the older two. Hence, you simply take the average of $E[x_1^2]$ and $E[x_2^2]$ when calculating new $E[x^2]$. Again, sorry for confusion. $\endgroup$ Aug 17, 2017 at 0:20
  • $\begingroup$ @ahmethungari I disagree: you give a formula for E[x^2] - E[x]^2 but you never say what "x" is, and it is not obvious. In particular, with your result, x cannot be the intuitive weighted average I used above. $\endgroup$
    – JulienD
    Aug 18, 2017 at 7:38
  • $\begingroup$ @JulienD if I take two singleton observations at -1 and 1, i.e. the parameters of the gaussians will be $n_1 = n_2 = 1$, $\sigma_2^2 = \sigma_1^2 = 0$, $\mu_1 = -1$, $\mu_2 = 1$, and join these together using your formula, I get $\sigma^2 = 0$, which is suspicious. $\endgroup$
    – pallly
    Oct 6, 2020 at 13:47
  • $\begingroup$ @JulienD: the formula variance of the new Gaussian distribution is missing power two of the mean mu. Please correct it. $\endgroup$
    – ngovanmao
    Nov 10, 2020 at 1:59
2
$\begingroup$

Given two multivariate normal distributions $N(\mu_0, \Sigma_0)$ and $N(\mu_1, \Sigma_1)$, the distribution you want is $N(\mu_f,\Sigma_f)$,

where

$\Sigma_f = \Sigma_1 (\Sigma_1 + \Sigma_0)^{-1}\Sigma_0$

$\mu_f = \Sigma_1 (\Sigma_1 + \Sigma_0)^{-1} \mu_0 + \Sigma_0 (\Sigma_1 + \Sigma_0)^{-1} \mu_1$

This works for any covariance matrices (no need to be diagonal).

$\endgroup$
1
$\begingroup$

As the answer of JulienD is good. But I would like to rewrite the final formulas in a better way and extend them to many random variables, and want to clarify that the approved answer is likely not correct.

First, I rewrite the answer of JulienD with few additional steps to give it a better form as follows:

  • Single variate: \begin{align} \mu &= E[x] = \frac{n_1\mu_1 + n_2\mu_2}{n_1 + n_2} \end{align}

\begin{align} \sigma^2 & = E[x^2] - E[x]^2 \\ & = \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2 + (n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} - \mu^2 \\ & = \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2}{(n_1+n_2)^2} + \frac{(n_1\mu_1 + n_2\mu_2)^2}{(n_1+n_2)^2} - \mu^2 \\ & = \frac{\sigma_1^2 n_1^2 + \sigma_2^2 n_2^2}{(n_1+n_2)^2} + \mu^2 - \mu^2 \\ & = \frac{n_1^2}{(n_1+n_2)^2}\sigma_1^2 + \frac{n_2^2}{(n_1+n_2)^2}\sigma_2^2 \end{align}

  • Multivariate: \begin{align} \Sigma &= E[XX^T] - E[X]E[X]^T \\ & = \frac{n_1^2 \Sigma_1 + n_2^2 \Sigma_2 + (n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T}{(n_1 + n_2)^2} - \mu\mu^T \\ &= \frac{n_1^2 \Sigma_1 + n_2^2 \Sigma_2}{(n_1 + n_2)^2} + \frac{(n_1\mu_1 + n_2\mu_2)(n_1\mu_1 + n_2\mu_2)^T}{(n_1 + n_2)^2} - \mu\mu^T \\ & = \frac{n_1^2 \Sigma_1 + n_2^2 \Sigma_2}{(n_1 + n_2)^2} + \mu\mu^T - \mu\mu^T \\ &= \frac{n_1^2}{(n_1 + n_2)^2} \Sigma_1 + \frac{n_2^2}{(n_1 + n_2)^2} \Sigma_2 \end{align}

Another approach to get a formula of variance or covariance matrix (for multivariate) is that we use the theorem of variance (which can find in many textbooks of statistics, e.g., Larry Wasserman, All of Statistics, theorem 3.20, page 52, or prove via definition as a similar way as JulienD did): $$ \mathbb{V}(X+Y) = \mathbb{V}(X) + \mathbb{V}(Y) + 2 \textbf{Cov}(X,Y) $$ More generally, for random variables $X_1, X_2,..., X_N$: $$ \mathbb{V}(\sum_{i=1}^{N}a_i X_i) = \sum_{i=1}^{N}a_i^2\mathbb{V}(X_i) + 2 \sum_{i=1}^{N-1}\sum_{j=i+1}^{N} a_i a_j \textbf{Cov}(X_i,X_j) $$

Consider $X_i \sim \mathcal{N}(\mu_i, \Sigma_i), i=1 \cdots N,$; and they are independent of each other. Hence, the covariances are zero, i.e., $\textbf{Cov}(X_i,X_j)=0.$ A newly aggregated these Gaussian distributions is defined as a weighted sum: $$X=\sum_{i=1}^{N} a_i X_i=\sum_{i=1}^{N} \frac{n_i}{\sum_{l=1}^{N}n_l} X_i,$$ where $a_i=\frac{n_i}{\sum_{l=1}^{N}n_l}$ and $\sum_{i=1}^N a_i = 1$.

So, we get the mean and covariance of the aggregated distribution as follows: $$ \mu = \sum_{i=1}^{N} a_i \mu_i $$ \begin{align} \Sigma &=\mathbb{V}(X)= \mathbb{V}(\sum_{i=1}^{N} a_i X_i)\\ &=\sum_{i=1}^{N}a_i^2\mathbb{V}(X_i) + 2 \sum_{i=1}^{N-1}\sum_{j=i+1}^{N} a_i a_j \textbf{Cov}(X_i,X_j) \\ & = \sum_{i=1}^{N}a_i^2 \Sigma_i + 0 \\ & = \sum_{i=1}^{N}a_i^2 \Sigma_i \end{align}

Now, look at the answer of ahmethungari, it seems the answer is not correct.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.