2
$\begingroup$

The following is from a textbook one bayesian stats. that I can't understand some deduction. It is relevant about multiple parameters to be estimated.

The jth observation in the ith group is denoted by $y_{ij}$,
where

$$(y_{ij}|\mu_i,\sigma)\sim N(\mu_i,\sigma^2) \quad j=1,2, \dots, n_i \quad i= 1,2, \dots, m$$

Also the $y_{ij}$ are independent from each other.

Suppose $\mu_i \sim N(\mu,\tau^2)$ and denote
$$\theta= (\mu, \log(\sigma),\log(\tau))$$ $$Y=\{y_{ij}: j=1,\dots, n_i, i=1,\dots, n\}$$ $$Z=(\mu_1,\dots, \mu_m)$$ $$n=n_1+n_2+\cdots +n_m$$

So $\theta$ is the unknown parameters interested. Take its prior distribution as $p(\theta) \propto \tau$. Then by Bayes rule, it is not difficult to get the posterior distribution:

$$p(Z,\theta|Y) \propto p(\theta) \prod\limits_{i = 1}^m {p(\mu_i|\mu,\tau)} \prod\limits_{i = 1}^m \prod\limits_{j = 1}^{n_i} {p(y_{ij}|\mu_i,\sigma)}$$

This is the place I can't understand. How to get this formula if No.3 formula is not correct in this thread: I am confused about Bayes' rule in MCMC

Could someone explain it in detail? If there are any excellent books that could help me, please list them.

$\endgroup$
5
  • 1
    $\begingroup$ You can use $\TeX$ on this site by enclosing formulas in dollar signs; single dollar signs for inline formulas and double dollar signs for displayed equations. You can see the source code for any math formatting you see on this site by right-clicking on it and selecting "Show Math As:TeX Commands". Here's a basic tutorial and quick reference. There's an "edit" link under the question. $\endgroup$
    – joriki
    Commented Oct 12, 2012 at 6:43
  • $\begingroup$ Thanks for your help. I have modified it just now. $\endgroup$
    – jerry
    Commented Oct 12, 2012 at 6:54
  • $\begingroup$ There are commands for all the Greek letters, e.g. \sigma yields $\sigma$. $\endgroup$
    – joriki
    Commented Oct 12, 2012 at 7:12
  • $\begingroup$ I have tidied up your post - I don't think I've introduced any errors, but you should check. $\endgroup$ Commented Oct 12, 2012 at 7:45
  • $\begingroup$ Thanks very much. I forget the Greek letters. $\endgroup$
    – jerry
    Commented Oct 12, 2012 at 8:14

1 Answer 1

2
$\begingroup$

AFAICT, the "trick" is that, by definition, $y_{ij}$ depends on $\mu$ and $\tau$ only via $\mu_i$. Thus, $$p(y_{ij}\mid\mu_i,\sigma) = p(y_{ij}\mid\mu_i,\sigma,\mu,\tau) = p(y_{ij}\mid\mu_i,\theta).$$

Similarly, $\mu_i$ does not depend on $\sigma$, so $$p(\mu_i\mid\mu,\tau) = p(\mu_i\mid\mu,\tau,\sigma) = p(\mu_i\mid\theta).$$ In particular, this means that we can rewrite your equation as

$$ \begin{aligned} p(Z,\theta\mid Y) \propto& p(\theta) \prod_{i = 1}^m p(\mu_i\mid\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}\mid\mu_i,\sigma) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i\mid\sigma,\mu,\tau) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}\mid\mu_i,\sigma,\mu,\tau) \\ =& p(\theta) \prod_{i = 1}^m p(\mu_i\mid\theta) \prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}\mid\mu_i,\theta) \\ =& p(\theta)\, p(Z\mid\theta)\, p(Y\mid Z,\theta) \\ =& p(Y,Z,\theta) \\ =& p(Z,\theta\mid Y)\, p(Y). \end{aligned} $$

$\endgroup$
9
  • $\begingroup$ The reason $p(y_{ij}|\mu_i,\sigma) = p(y_{ij}|\mu_i,\sigma,\mu,\tau)$ is if $\mu_i$ is fixed in condition, then $\mu$ and $\tau$ are redundant due to the distribution of ${y_{ij}}$ is known with fixed $\mu_i$ and $\sigma$. Am I right to understand your point? There are so many places using this "trick". Thanks very much. $\endgroup$
    – jerry
    Commented Oct 12, 2012 at 9:18
  • $\begingroup$ I have the following question for your deduction. How to get $\prod_{i = 1}^m \prod_{j = 1}^{n_i} p(y_{ij}|\mu_i,\theta) =p(Y|Z,\theta)$? Is there the equality that $p(X1|Y)*p(X2|Z)=p(X1,X2|Y,Z)$ $\endgroup$
    – jerry
    Commented Oct 12, 2012 at 9:25
  • 1
    $\begingroup$ For you first question, I think I agree, if I understand you correctly. What really matters is that $y_{ij}$, by definition, depends only on $\mu_i$ and $\sigma$, and is therefore conditionally independent of anything else. As for your second question, indeed, $p(X_1|Y)\,p(X_2|Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\, Y$ and $X_2\perp Y\,|\,Z$, in which case $p(X_1|Y)=p(X_1|Y,Z)$ and $p(X_2|Z)=p(X_2|Y,Z)$. In particular, here, we know that, by definition, $y_{ij}\perp\mu_k$ whenever $i\ne k$. $\endgroup$ Commented Oct 12, 2012 at 11:44
  • 1
    $\begingroup$ By definition, $A\perp B\,|\,C$ iff $p(A,B|C)=p(A|C)\,p(B|C)$. Divide both sides by $p(B|C)$ to get $p(A|B,C)=p(A|C)$. Apply with appropriate substitutions to get $p(X_1|Y)\,p(X_2|Z)=p(X_1|Y,Z)\,p(X_2|Y,Z)=p(X_1,X_2|Y,Z)$ if $X_1\perp Z\,|\,Y$ and $X_2\perp Y\,|\,Z$. $\endgroup$ Commented Oct 12, 2012 at 14:18
  • 1
    $\begingroup$ @IlmariKaronen - More explicitly, sufficient conditions for $\ \ p(X_1,X_2|Y,Z)=p(X_1|Y)p(X_2|Z)\ \ $ are $\ \ (X_1\perp Z)\mid Y,$ $\ \ (X_2\perp Y)\mid Z,\ \ $ and $\ \ (X_1\perp X_2)\mid (Y,Z).\ \ $ (The latter condition was tacitly assumed in the above comments.) $\endgroup$
    – r.e.s.
    Commented Nov 24, 2017 at 3:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .