Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For mixture of multivariate Bernoulli distribution we have that,

$$p(x|\mu,\pi) =\Sigma_{k=1}^{K}\pi_kp(x|\mu_k)$$ where $$p(x|\mu_k) = \prod_{i=1}^{D}\mu_{ki}^{x_i}(1-\mu_{ki})^{1-x_i}$$

I read it from the book that

$$E[x] = \Sigma_{i=1}^{K}\pi_k\mu_k$$ $$cov[x] = \Sigma_{k=1}^{K}\pi_k(\Sigma_k+\mu_k\mu_k^T) - E[x]E[x]^T$$

The mean is trivial to prove, however I can't find proof for the covariance and I don't know how to prove it.

Can anyone help?

share|cite|improve this question
up vote 1 down vote accepted

You know that, in general, $cov[x] = E[x x^T] - E[x]E[x]^T$ and you want to compute the first term. We can use the property: $E[g(X)] = E [ E(g(X)|k]]$

But $E(x x^T | k)$ (i.e., fixing the population 'k' index of the mixing), is $\Sigma_k+\mu_k\mu_k^T$ (don't let the $k$ subindexes confuse you: they are the first and seconds moments of $x$, for a fixed population index $k$).

Now, the above is a function ok $k$, and its expectation is $\sum_{k=1}^K \pi_k (\Sigma_k+\mu_k\mu_k^T)$

Notice that this result is not really connected to the Bernoulli distribution, it's valid in general.

share|cite|improve this answer
Hi, can I know why $$E(xx^T|k) = \Sigma_k+\mu_k\mu_k^T$$ I think that's the thing that confuses me – Jing Nov 30 '12 at 4:55
@Jing: It's just the same first equation of my answer, rearranged, and applied not to $x$ (the mixed variable) but to $x_k$ (one of the $K$ components of the mixture) – leonbloy Nov 30 '12 at 11:50

\begin{equation} \begin{split} cov[x]&=E[xx^{T}]−E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}E[xx^{T}|k] − E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}(E[xx^{T}|k]-E[x|k]E[x|k]^{T}+E[x|k]E[x|k]^{T}) − E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}(E[(x-E[x])^{2}|k]+E[x|k]E[x|k]^{T})− E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}(Σ_{k}+μ_{k}μ^{T}_{k})− E[x]E[x]^{T} \end{split} \end{equation} Where $Σ_{k} = \text{diag}(μ_{ki}(1-μ_{ki}))$ (i=1,...,D) if $\{x_{i}\}_{i=1}^{D}$ are pairwise independent variables.

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.