# A confusing excersice about Bayes' rule

The following is from a textbook one bayesian stats. that I can't understand some deduction. It is relevant about multiple parameters to be estimated.

The jth observation in the ith group is denoted by $y_{ij}$,
where

$$(y_{ij}|\mu_i,\sigma)\sim N(\mu_i,\sigma^2) \quad j=1,2, \dots, n_i \quad i= 1,2, \dots, m$$

Also the $y_{ij}$ are independent from each other.

Suppose $\mu_i \sim N(\mu,\tau^2)$ and denote
$$\theta= (\mu, \log(\sigma),\log(\tau))$$ $$Y=\{y_{ij}: j=1,\dots, n_i, i=1,\dots, n\}$$ $$Z=(\mu_1,\dots, \mu_m)$$ $$n=n_1+n_2+\cdots +n_m$$

So $\theta$ is the unknown parameters interested. Take its prior distribution as $p(\theta) \propto \tau$. Then by Bayes rule, it is not difficult to get the posterior distribution:

$$p(Z,\theta|Y) \propto p(\theta) \prod\limits_{i = 1}^m {p(\mu_i|\mu,\tau)} \prod\limits_{i = 1}^m \prod\limits_{j = 1}^{n_i} {p(y_{ij}|\mu_i,\sigma)}$$

This is the place I can't understand. How to get this formula if No.3 formula is not correct in this thread: I am confused about Bayes' rule in MCMC

Could someone explain it in detail? If there are any excellent books that could help me, please list them.

-
You can use $\TeX$ on this site by enclosing formulas in dollar signs; single dollar signs for inline formulas and double dollar signs for displayed equations. You can see the source code for any math formatting you see on this site by right-clicking on it and selecting "Show Math As:TeX Commands". Here's a basic tutorial and quick reference. There's an "edit" link under the question. –  joriki Oct 12 '12 at 6:43
Thanks for your help. I have modified it just now. –  jerry Oct 12 '12 at 6:54
There are commands for all the Greek letters, e.g. \sigma yields $\sigma$. –  joriki Oct 12 '12 at 7:12
I have tidied up your post - I don't think I've introduced any errors, but you should check. –  Chris Taylor Oct 12 '12 at 7:45
Thanks very much. I forget the Greek letters. –  jerry Oct 12 '12 at 8:14

AFAICT, the "trick" is that, by definition, $y_{ij}$ depends on $\mu$ and $\tau$ only via $\mu_i$. Thus, $$p(y_{ij}|\mu_i,\sigma) = p(y_{ij}|\mu_i,\sigma,\mu,\tau) = p(y_{ij}|\mu_i,\theta).$$
Similarly, $\mu_i$ does not depend on $\sigma$, so $$p(\mu_i|\mu,\tau) = p(\mu_i|\mu,\tau,\sigma) = p(\mu_i|\theta).$$ In particular, this means that we can rewrite your equation as
\begin{aligned} p(Z,\theta|Y) \propto& p(\theta) \prod_{i = 1}^m p(\mu_i|\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\sigma) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i|\sigma,\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\sigma,\mu,\tau) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i|\theta) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\theta) \\ =& p(\theta)\, p(Z|\theta)\, p(Y|Z,\theta) \\ =& p(Y,Z,\theta) \\ =& p(Z,\theta|Y)\, p(Y). \end{aligned}
The reason $p(y_{ij}|\mu_i,\sigma) = p(y_{ij}|\mu_i,\sigma,\mu,\tau)$ is if $\mu_i$ is fixed in condition, then $\mu$ and $\tau$ are redundant due to the distribution of ${y_{ij}}$ is known with fixed $\mu_i$ and $\sigma$. Am I right to understand your point? There are so many places using this "trick". Thanks very much. –  jerry Oct 12 '12 at 9:18
I have the following question for your deduction. How to get $\prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\theta) =p(Y|Z,\theta)$? Is there the equality that $p(X1|Y)*p(X2|Z)=p(X1,X2|Y,Z)$ –  jerry Oct 12 '12 at 9:25
For you first question, I think I agree, if I understand you correctly. What really matters is that $y_{ij}$, by definition, depends only on $\mu_i$ and $\sigma$, and is therefore conditionally independent of anything else. As for your second question, indeed, $p(X_1|Y)\,p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\, Y$ and $X_2\perp Y\,|\,Z$, in which case $p(X_1|Y)=p(X_1|Y,Z)$ and $p(X_2|Z)=p(X_2|Y,Z)$. In particular, here, we know that, by definition, $y_{ij}\perp\mu_k$ whenever $i\ne k$. –  Ilmari Karonen Oct 12 '12 at 11:44
Your answer helps. I appreciate it very much. I regret that my reputation is not enough to support your answer. But I will come back after accumulation is done. Another question: how to deduct $p(X_1|Y)p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z|Y$ and $X_2\perp Y|Z$? I know the meaning of $X_1\perp X_2$ but donot know the conditional form. –  jerry Oct 12 '12 at 12:47
By definition, $A\perp B\,|\,C$ iff $p(A,B|C)=p(A|C)\,p(B|C)$. Divide both sides by $p(B|C)$ to get $p(A|B,C)=p(A|C)$. Apply with appropriate substitutions to get $p(X_1|Y)\,p(X_2|Z)=p(X_1|Y,Z)\,p(X_2|Y,Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\,Y$ and $X_2\perp Y\,|\,Z$. –  Ilmari Karonen Oct 12 '12 at 14:18