# Covariance of Binomials

I'm doing some basic error, but I just can't see where...

Let $X_i\sim \mathrm{Bin}(\theta_i,n)$, and $X_j\sim \mathrm{Bin}(\theta_j,n)$

I want to find $\mathrm{Cov}(X_i,X_j)$.

So,

$E(X_iX_j)=E(E(X_iX_j|X_i=x_i))=E(X_i n \theta_j)=n\theta_i n\theta_j$. But this isn't correct is it? Why?

• We need information about any possible relationship between $X_i$ and $X_j$. If they are independent, the covariance is $0$. – André Nicolas Jul 9 '14 at 21:53
• @AndréNicolas, thanks for your interest in this question. The only thing I know is that $\mathbf{X}=(X_1,...,X_k)$, where $\mathbf{X} \sim Multi (n,\mathbf{\Theta})$ – An old man in the sea. Jul 9 '14 at 21:54
• So, the marginal distribution of a multinomial is a binomial. – An old man in the sea. Jul 9 '14 at 22:07
• If you are interested in the distribution of the number of Type $j$, then yes. – André Nicolas Jul 9 '14 at 22:10
• The binomials are in this case not independent. – André Nicolas Jul 9 '14 at 22:20

Recall that $X=(X_i)_i$ multinomial $(n,(\theta_i)_i)$ with $\sum\limits_i\theta_i=1$ can be realized as $$X_i=\sum\limits_{k=1}^n\mathbf 1_{U_k=i},$$ for every $i$, where $(U_k)$ is i.i.d. with $P(U_k=i)=\theta_i$ for every $k$ and every $i$. Thus, for every $i\ne j$, $$X_iX_j=\sum_k\mathbf 1_{U_k=i}\mathbf 1_{U_k=j}+\sum_{k\ne\ell}\mathbf 1_{U_k=i}\mathbf 1_{U_\ell=j}.$$ Since $[U_k=i]\cap[U_k=j]=\varnothing$ for every $k$ and the events $[U_k=i]$ and $[U_\ell=j]$ are independent for every $k\ne\ell$, this yields $$E(X_iX_j)=\sum_{k\ne\ell}P(U_k=i)P(U_\ell=j)=n(n-1)\theta_i\theta_j.$$ Since $E(X_i)=n\theta_i$ and $E(X_j)=n\theta_j$, one gets $$\mathrm{Cov}(X_i,X_j)=n(n-1)\theta_i\theta_j-n\theta_in\theta_j=-n\theta_i\theta_j.$$

The covariance should be $E(X_iX_j - E(X_i)E(X_j))$, which in your calculations could be $0$ when $i \not = j$.

But you seem to have said $E(X_j|X_i=x_i)= n\theta_j$. Are you sure? What is $E(X_j|X_i=n)$?

• many thanks for your pedagogical perspective. Afterwards I decided to go with the moment generating function... Just to be sure, $E(X_j|X_i=n)=0$, right? – An old man in the sea. Jul 10 '14 at 10:17
• @An old man in the sea: Yes - if everything is type $j$ then nothing can be of type $i$. – Henry Jul 10 '14 at 10:21

From a comment by the OP, the real question is as follows. We repeat an experiment independently $n$ times. The possible outcomes are $A_1, A_2, \dots, A_q$ with probabilities $\theta_1,\theta_2,\dots,\theta_q$. Let $X_i$ be the number of Type $A_i$ outcomes in the $n$ trials. We want $\text{Cov}(X_i,X_j)$. Informally, the covariance is negative: if $A_i$ occurs lotsa times, then probably $X_j$ will not occur many times.

So we want $E(X_iX_j)-E(X_i)E(X_j)$. Te expectations $E(X_i)$ and $E(X_j)$ are easy. We need to find $E(X_iX_j)$.

Let $U_k=1$ if on the $k$-th trial we get $A_i$. Let $V_k$ be defined similarly for $A_j$. We want $E((U_1+\cdots+U_n)(V_1+\cdots +V_n))$. By the linearity of expectation, and the fact that $U_sV_s=0$ and $\Pr(U_sV_t=1)=\theta_i\theta_j$ if $s\ne t$, we get that $E((U_1+\cdots+U_n)(V_1+\cdots +V_n))=n(n-1) \theta_i\theta_j$.

But $E(X_i)=n\theta_i$, with a similar expression for $E(X_j)$. Subtract. We get the desired result.

• Many thanks for your help. – An old man in the sea. Jul 10 '14 at 10:19
• You are welcome. – André Nicolas Jul 10 '14 at 12:09