Covariance of Binomials I'm doing some basic error, but I just can't see where...
Let $X_i\sim \mathrm{Bin}(\theta_i,n)$, and $X_j\sim \mathrm{Bin}(\theta_j,n)$
I want to find $\mathrm{Cov}(X_i,X_j)$.
So,
$E(X_iX_j)=E(E(X_iX_j|X_i=x_i))=E(X_i n \theta_j)=n\theta_i n\theta_j$. But this isn't correct is it? Why?
 A: The covariance should be $E(X_iX_j - E(X_i)E(X_j))$, which in your calculations could be $0$ when $i \not = j$.
But you seem to have said $E(X_j|X_i=x_i)= n\theta_j$.  Are you sure?  What is  $E(X_j|X_i=n)$?
A: Recall that $X=(X_i)_i$ multinomial $(n,(\theta_i)_i)$ with $\sum\limits_i\theta_i=1$ can be realized as $$X_i=\sum\limits_{k=1}^n\mathbf 1_{U_k=i},$$ for every $i$, where $(U_k)$ is i.i.d. with $P(U_k=i)=\theta_i$ for every $k$ and every $i$. Thus, for every $i\ne j$,
$$
X_iX_j=\sum_k\mathbf 1_{U_k=i}\mathbf 1_{U_k=j}+\sum_{k\ne\ell}\mathbf 1_{U_k=i}\mathbf 1_{U_\ell=j}.$$
Since $[U_k=i]\cap[U_k=j]=\varnothing$ for every $k$ and the events $[U_k=i]$ and $[U_\ell=j]$ are independent for every $k\ne\ell$, this yields
$$
E(X_iX_j)=\sum_{k\ne\ell}P(U_k=i)P(U_\ell=j)=n(n-1)\theta_i\theta_j.
$$
Since $E(X_i)=n\theta_i$ and $E(X_j)=n\theta_j$, one gets
$$
\mathrm{Cov}(X_i,X_j)=n(n-1)\theta_i\theta_j-n\theta_in\theta_j=-n\theta_i\theta_j.$$
A: From a comment by the OP, the real question is as follows. We repeat an experiment independently $n$ times. The possible outcomes are $A_1, A_2, \dots, A_q$ with probabilities $\theta_1,\theta_2,\dots,\theta_q$. Let $X_i$ be the number of Type $A_i$ outcomes in the $n$ trials. We want $\text{Cov}(X_i,X_j)$. Informally, the covariance is negative: if $A_i$ occurs lotsa times, then probably $X_j$ will not occur many times.  
So we want $E(X_iX_j)-E(X_i)E(X_j)$.  Te expectations $E(X_i)$ and $E(X_j)$ are easy. We need to find $E(X_iX_j)$.
Let $U_k=1$ if on the $k$-th trial we get $A_i$. Let $V_k$ be defined similarly for $A_j$. We want $E((U_1+\cdots+U_n)(V_1+\cdots +V_n))$. By the linearity of expectation, and the fact that $U_sV_s=0$ and $\Pr(U_sV_t=1)=\theta_i\theta_j$ if $s\ne t$, we get that $E((U_1+\cdots+U_n)(V_1+\cdots +V_n))=n(n-1) \theta_i\theta_j$.
But $E(X_i)=n\theta_i$, with a similar expression for $E(X_j)$. Subtract. We get the desired result.
