Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

The following is from a textbook one bayesian stats. that I can't understand some deduction. It is relevant about multiple parameters to be estimated.

The jth observation in the ith group is denoted by $y_{ij}$,

$$(y_{ij}|\mu_i,\sigma)\sim N(\mu_i,\sigma^2) \quad j=1,2, \dots, n_i \quad i= 1,2, \dots, m$$

Also the $y_{ij}$ are independent from each other.

Suppose $\mu_i \sim N(\mu,\tau^2)$ and denote
$$\theta= (\mu, \log(\sigma),\log(\tau))$$ $$Y=\{y_{ij}: j=1,\dots, n_i, i=1,\dots, n\}$$ $$Z=(\mu_1,\dots, \mu_m)$$ $$n=n_1+n_2+\cdots +n_m$$

So $\theta$ is the unknown parameters interested. Take its prior distribution as $p(\theta) \propto \tau$. Then by Bayes rule, it is not difficult to get the posterior distribution:

$$p(Z,\theta|Y) \propto p(\theta) \prod\limits_{i = 1}^m {p(\mu_i|\mu,\tau)} \prod\limits_{i = 1}^m \prod\limits_{j = 1}^{n_i} {p(y_{ij}|\mu_i,\sigma)}$$

This is the place I can't understand. How to get this formula if No.3 formula is not correct in this thread: I am confused about Bayes' rule in MCMC

Could someone explain it in detail? If there are any excellent books that could help me, please list them.

share|improve this question
You can use $\TeX$ on this site by enclosing formulas in dollar signs; single dollar signs for inline formulas and double dollar signs for displayed equations. You can see the source code for any math formatting you see on this site by right-clicking on it and selecting "Show Math As:TeX Commands". Here's a basic tutorial and quick reference. There's an "edit" link under the question. –  joriki Oct 12 '12 at 6:43
Thanks for your help. I have modified it just now. –  jerry Oct 12 '12 at 6:54
There are commands for all the Greek letters, e.g. \sigma yields $\sigma$. –  joriki Oct 12 '12 at 7:12
I have tidied up your post - I don't think I've introduced any errors, but you should check. –  Chris Taylor Oct 12 '12 at 7:45
Thanks very much. I forget the Greek letters. –  jerry Oct 12 '12 at 8:14
add comment

1 Answer

up vote 1 down vote accepted

AFAICT, the "trick" is that, by definition, $y_{ij}$ depends on $\mu$ and $\tau$ only via $\mu_i$. Thus, $$p(y_{ij}|\mu_i,\sigma) = p(y_{ij}|\mu_i,\sigma,\mu,\tau) = p(y_{ij}|\mu_i,\theta).$$

Similarly, $\mu_i$ does not depend on $\sigma$, so $$p(\mu_i|\mu,\tau) = p(\mu_i|\mu,\tau,\sigma) = p(\mu_i|\theta).$$ In particular, this means that we can rewrite your equation as

$$ \begin{aligned} p(Z,\theta|Y) \propto& p(\theta) \prod_{i = 1}^m p(\mu_i|\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\sigma) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i|\sigma,\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\sigma,\mu,\tau) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i|\theta) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\theta) \\ =& p(\theta)\, p(Z|\theta)\, p(Y|Z,\theta) \\ =& p(Y,Z,\theta) \\ =& p(Z,\theta|Y)\, p(Y). \end{aligned} $$

share|improve this answer
The reason $p(y_{ij}|\mu_i,\sigma) = p(y_{ij}|\mu_i,\sigma,\mu,\tau)$ is if $\mu_i$ is fixed in condition, then $\mu$ and $\tau$ are redundant due to the distribution of ${y_{ij}}$ is known with fixed $\mu_i$ and $\sigma$. Am I right to understand your point? There are so many places using this "trick". Thanks very much. –  jerry Oct 12 '12 at 9:18
I have the following question for your deduction. How to get $\prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\theta) =p(Y|Z,\theta)$? Is there the equality that $p(X1|Y)*p(X2|Z)=p(X1,X2|Y,Z)$ –  jerry Oct 12 '12 at 9:25
For you first question, I think I agree, if I understand you correctly. What really matters is that $y_{ij}$, by definition, depends only on $\mu_i$ and $\sigma$, and is therefore conditionally independent of anything else. As for your second question, indeed, $p(X_1|Y)\,p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\, Y$ and $X_2\perp Y\,|\,Z$, in which case $p(X_1|Y)=p(X_1|Y,Z)$ and $p(X_2|Z)=p(X_2|Y,Z)$. In particular, here, we know that, by definition, $y_{ij}\perp\mu_k$ whenever $i\ne k$. –  Ilmari Karonen Oct 12 '12 at 11:44
Your answer helps. I appreciate it very much. I regret that my reputation is not enough to support your answer. But I will come back after accumulation is done. Another question: how to deduct $p(X_1|Y)p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z|Y$ and $X_2\perp Y|Z$? I know the meaning of $X_1\perp X_2$ but donot know the conditional form. –  jerry Oct 12 '12 at 12:47
By definition, $A\perp B\,|\,C$ iff $p(A,B|C)=p(A|C)\,p(B|C)$. Divide both sides by $p(B|C)$ to get $p(A|B,C)=p(A|C)$. Apply with appropriate substitutions to get $p(X_1|Y)\,p(X_2|Z)=p(X_1|Y,Z)\,p(X_2|Y,Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\,Y$ and $X_2\perp Y\,|\,Z$. –  Ilmari Karonen Oct 12 '12 at 14:18
show 3 more comments

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.