19
$\begingroup$

If $(X_1,\cdots, X_n)$ is a vector with multinomial distribution, proof that $\text{Cov}(X_i,X_j)=-rp_ip_j$, $i\neq j$ where $r$ is the number of trials of the experiment, $p_i$ is the probability of success for the variable $X_i$. $$fdp=f(x_1,...x_n)={r!\over{x_1!x_2!\cdots x_n!}}p_1^{x_1}\cdots p_n^{x_n} $$ if $ x_1+x_2+\cdots +x_n=r$

I'm trying to use the property: $\text{Cov}(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j]$ and find that $E[X_i]=rp_i$, but I don´t know the efficient way to calculate $E[X_iX_j].$

$\endgroup$
3
  • $\begingroup$ What is $E(X_iX_j)$ for $r=1$? $\endgroup$
    – A.S.
    Feb 24, 2016 at 1:37
  • $\begingroup$ r=x1+x2+...+xn, if r=1 then x1+x2+...+xn=1 $\endgroup$
    – User 2014
    Feb 24, 2016 at 1:48
  • 1
    $\begingroup$ As what A.S. hinted, one common trick is to express $X_i = \sum_{k=1}^r Y_{i,k}, X_j = \sum_{l=1}^r Y_{j,l}$ and use linearity of covariance. By independence across different multinomial trials, you only left the calculate the case with $Cov[Y_{i,k}, Y_{j, k}]$. But those $Y$ are indicators only (i.e. the $r = 1$ case mentioned by A.S.) which is easy to calculate. $\endgroup$
    – BGM
    Feb 24, 2016 at 3:43

1 Answer 1

28
$\begingroup$

We can use indicator random variables to help simplify the covariance expression. We can interpret the problem as $r$ independent rolls of an $n$ sided die. Let $X_i$ be the number of rolls that result in side $i$ facing up, and let $I_{k}^{(i)}$ be an indicator equal to $1$ when roll $k$ is equal to $i$ and $0$ otherwise. Then, we can express $X_i$ and $X_j$ as follows:

$$\begin{equation} X_i = \sum_{k=1}^{r} I_{k}^{(i)}~~~\mathrm{and}~~~X_j = \sum_{k=1}^{r} I_{k}^{(j)} \end{equation}$$

Let's re-write the covariance using indicators: $$\begin{equation} \mathrm{Cov}(X_i,X_j) = E[X_i X_j] - E[X_i]E[X_j] \end{equation}$$ Let's compute the first term: $$\begin{eqnarray} E[X_i X_j] &=& E\bigg[(\sum_{k=1}^{r}I_{k}^{(i)}) (\sum_{l=1}^{r}I_{l}^{(j)})\bigg] = \sum_{k=l}E\big[I_{k}^{(i)}I_{l}^{(j)}\big] + \sum_{k\neq l}E\big[I_{k}^{(i)}I_{l}^{(j)}\big] = \\ &=& 0 + \sum_{k\neq l}E\big[I_{k}^{(i)}\big] E\big[I_{l}^{(j)}\big] = \sum_{k\neq l} p_i p_j = (r^2 - r)p_i p_j \end{eqnarray}$$ where we expanded the product of sums, used linearity of expectation and the fact that when $k=l$ we can't simultaneously roll $i$ and $j$ on the same trial $k=l$ (making the product of indicators zero) Finally we applied independence of rolls that enabled us to write it as a product of probabilities. Let's compute the remaining term: $$\begin{equation} E[X_i] = E[\sum_{k=1}^{r}I_{k}^{(i)}] = \sum_{k=1}^{r}E[I_{k}^{(i)}] = rp_i \end{equation}$$ Therefore, the covariance equals: $$\begin{equation} \mathrm{Cov}(X_i,X_j) = E[X_i X_j] - E[X_i]E[X_j] = (r^2-r)p_ip_j - r^2p_ip_j = -r p_i p_j \end{equation}$$ Notice that $\mathrm{Cov}(X_i, X_j) = -r p_i p_j < 0$ is negative, this makes sense intuitively since for a fixed number of rolls $r$, if we roll many outcomes $i$, this reduces the number of possible outcomes $j$, and therefore $X_i$ and $X_j$ are negatively correlated!

$\endgroup$
4
  • 1
    $\begingroup$ @amWhy The statement "when $k=l$ we can't simultaneously roll $i$ and $j$ on the same trial $k=l$ (making the product of indicators zero) " was true only for $i\neq j$, for $i=j$, $I_k^{(i)} I_l^{(j)}$ was the multiplication of same sequence of $0$ or $1$, thus equal to $I_k^{(i)}$ for diagonal $k=l$. $\endgroup$ Sep 3, 2020 at 17:54
  • $\begingroup$ @amWhy For $i=j$, $$\sum_{k=l}E\big[I_{k}^{(i)}I_{l}^{(j)}\big] = \sum_{k=l}E\big[I_{k}^{(i)}I_{l}^{(i)}\big]=\sum_{k}E\big[I_{k}^{(i)}\big] = r p_i$$. $\endgroup$ Sep 3, 2020 at 17:56
  • $\begingroup$ @amWhy Thus the original statement $Cov(X_i,X_j)=E[X_iX_j]-E[X_i]E[X_j] = rp_i \delta^i_j +(r^2-r)p_ip_j - r^2p_ip_j =rp_i \delta^i_j -rp_ip_j$. $\endgroup$ Sep 3, 2020 at 17:57
  • $\begingroup$ Instead of indicators, can we express $X_i$ as the sum of $N$ Bernoulli random variables? (thus rendering $X_i$ a binomial random variable) $\endgroup$ Oct 20, 2021 at 17:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .