# What is the covariance of mixture Bernoulli distribution?

For mixture of multivariate Bernoulli distribution we have that,

$$p(x|\mu,\pi) =\sum_{k=1}^{K}\pi_kp(x|\mu_k)$$ where $$p(x|\mu_k) = \prod_{i=1}^{D}\mu_{ki}^{x_i}(1-\mu_{ki})^{1-x_i}$$

I read it from the book that

$$E[x] = \sum_{i=1}^{K}\pi_k\mu_k$$ $$\operatorname{cov}[x] = \sum_{k=1}^{K}\pi_k(\Sigma_k+\mu_k\mu_k^T) - E[x]E[x]^T$$

The mean is trivial to prove, however I can't find proof for the covariance and I don't know how to prove it.

Can anyone help?

You know that, in general, $$\operatorname{cov}[x] = E[x x^T] - E[x]E[x]^T$$ and you want to compute the first term. We can use the property: $$E[g(X)] = E [ E(g(X)|k]]$$

But $$E(x x^T | k)$$ (i.e., fixing the population 'k' index of the mixing), is $$\Sigma_k+\mu_k\mu_k^T$$ (don't let the $$k$$ subindexes confuse you: they are the first and seconds moments of $$x$$, for a fixed population index $$k$$).

Now, the above is a function ok $$k$$, and its expectation is $$\sum_{k=1}^K \pi_k (\Sigma_k+\mu_k\mu_k^T)$$

Notice that this result is not really connected to the Bernoulli distribution, it's valid in general.

• Hi, can I know why $$E(xx^T|k) = \Sigma_k+\mu_k\mu_k^T$$ I think that's the thing that confuses me
– Jing
Nov 30, 2012 at 4:55
• @Jing: It's just the same first equation of my answer, rearranged, and applied not to $x$ (the mixed variable) but to $x_k$ (one of the $K$ components of the mixture) Nov 30, 2012 at 11:50

$$\begin{equation} \begin{split} \operatorname{cov}[x]&=E[xx^{T}]−E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}E[xx^{T}|k] − E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}(E[xx^{T}|k]-E[x|k]E[x|k]^{T}+E[x|k]E[x|k]^{T}) − E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}(E[(x-E[x])^{2}|k]+E[x|k]E[x|k]^{T})− E[x]E[x]^{T} \\ &=\sum^{K}_{k=1}π_{k}(Σ_{k}+μ_{k}μ^{T}_{k})− E[x]E[x]^{T} \end{split} \end{equation}$$ Where $$Σ_{k} = \text{diag}(μ_{ki}(1-μ_{ki}))$$ ($$i=1,...,D$$) if $$\{x_{i}\}_{i=1}^{D}$$ are pairwise independent variables.