# Show that the multinomial distribution has covariances ${\rm Cov}(X_i,X_j)=-r p_i p_j$

If $(X_1,\cdots, X_n)$ is a vector with multinomial distribution, proof that $\text{Cov}(X_i,X_j)=-rp_ip_j$, $i\neq j$ where $r$ is the number of trials of the experiment, $p_i$ is the probability of success for the variable $X_i$. $$fdp=f(x_1,...x_n)={r!\over{x_1!x_2!\cdots x_n!}}p_1^{x_1}\cdots p_n^{x_n}$$ if $x_1+x_2+\cdots +x_n=r$

I'm trying to use the property: $\text{Cov}(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j]$ and find that $E[X_i]=rp_i$, but I don´t know the efficient way to calculate $E[X_iX_j].$

• What is $E(X_iX_j)$ for $r=1$?
– A.S.
Feb 24, 2016 at 1:37
• r=x1+x2+...+xn, if r=1 then x1+x2+...+xn=1 Feb 24, 2016 at 1:48
• As what A.S. hinted, one common trick is to express $X_i = \sum_{k=1}^r Y_{i,k}, X_j = \sum_{l=1}^r Y_{j,l}$ and use linearity of covariance. By independence across different multinomial trials, you only left the calculate the case with $Cov[Y_{i,k}, Y_{j, k}]$. But those $Y$ are indicators only (i.e. the $r = 1$ case mentioned by A.S.) which is easy to calculate.
– BGM
Feb 24, 2016 at 3:43

## 1 Answer

We can use indicator random variables to help simplify the covariance expression. We can interpret the problem as $$r$$ independent rolls of an $$n$$ sided die. Let $$X_i$$ be the number of rolls that result in side $$i$$ facing up, and let $$I_{k}^{(i)}$$ be an indicator equal to $$1$$ when roll $$k$$ is equal to $$i$$ and $$0$$ otherwise. Then, we can express $$X_i$$ and $$X_j$$ as follows:

$$$$X_i = \sum_{k=1}^{r} I_{k}^{(i)}~~~\mathrm{and}~~~X_j = \sum_{k=1}^{r} I_{k}^{(j)}$$$$

Let's re-write the covariance using indicators: $$$$\mathrm{Cov}(X_i,X_j) = E[X_i X_j] - E[X_i]E[X_j]$$$$ Let's compute the first term: $$\begin{eqnarray} E[X_i X_j] &=& E\bigg[(\sum_{k=1}^{r}I_{k}^{(i)}) (\sum_{l=1}^{r}I_{l}^{(j)})\bigg] = \sum_{k=l}E\big[I_{k}^{(i)}I_{l}^{(j)}\big] + \sum_{k\neq l}E\big[I_{k}^{(i)}I_{l}^{(j)}\big] = \\ &=& 0 + \sum_{k\neq l}E\big[I_{k}^{(i)}\big] E\big[I_{l}^{(j)}\big] = \sum_{k\neq l} p_i p_j = (r^2 - r)p_i p_j \end{eqnarray}$$ where we expanded the product of sums, used linearity of expectation and the fact that when $$k=l$$ we can't simultaneously roll $$i$$ and $$j$$ on the same trial $$k=l$$ (making the product of indicators zero) Finally we applied independence of rolls that enabled us to write it as a product of probabilities. Let's compute the remaining term: $$$$E[X_i] = E[\sum_{k=1}^{r}I_{k}^{(i)}] = \sum_{k=1}^{r}E[I_{k}^{(i)}] = rp_i$$$$ Therefore, the covariance equals: $$$$\mathrm{Cov}(X_i,X_j) = E[X_i X_j] - E[X_i]E[X_j] = (r^2-r)p_ip_j - r^2p_ip_j = -r p_i p_j$$$$ Notice that $$\mathrm{Cov}(X_i, X_j) = -r p_i p_j < 0$$ is negative, this makes sense intuitively since for a fixed number of rolls $$r$$, if we roll many outcomes $$i$$, this reduces the number of possible outcomes $$j$$, and therefore $$X_i$$ and $$X_j$$ are negatively correlated!

• @amWhy The statement "when $k=l$ we can't simultaneously roll $i$ and $j$ on the same trial $k=l$ (making the product of indicators zero) " was true only for $i\neq j$, for $i=j$, $I_k^{(i)} I_l^{(j)}$ was the multiplication of same sequence of $0$ or $1$, thus equal to $I_k^{(i)}$ for diagonal $k=l$. Sep 3, 2020 at 17:54
• @amWhy For $i=j$, $$\sum_{k=l}E\big[I_{k}^{(i)}I_{l}^{(j)}\big] = \sum_{k=l}E\big[I_{k}^{(i)}I_{l}^{(i)}\big]=\sum_{k}E\big[I_{k}^{(i)}\big] = r p_i$$. Sep 3, 2020 at 17:56
• @amWhy Thus the original statement $Cov(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j] = rp_i \delta^i_j +(r^2-r)p_ip_j - r^2p_ip_j =rp_i \delta^i_j -rp_ip_j$. Sep 3, 2020 at 17:57
• Instead of indicators, can we express $X_i$ as the sum of $N$ Bernoulli random variables? (thus rendering $X_i$ a binomial random variable) Oct 20, 2021 at 17:09