Take the 2-minute tour ×
Mathematics Stack Exchange is a question and answer site for people studying math at any level and professionals in related fields. It's 100% free, no registration required.

I've been trying to understand the motivation for the use of the Jeffreys prior in Bayesian statistics. Most texts I've read online make some comment to the effect that the Jeffreys prior is "invariant with respect to transformations of the parameters", and then go on to state its definition in terms of the Fisher information matrix without further motivation. However, none of them then go on to show that such a prior is indeed invariant, or even to properly define what was meant by "invariant" in the first place.

I like to understand things by approaching the simplest example first, so I'm interested in the case of a binomial trial, i.e. the case where the support is $\{1,2\}$. In this case the Jeffreys prior is given by $$ \rho(\theta) = \frac{1}{\pi\sqrt{\theta(1-\theta)}}, \qquad\qquad(i) $$ where $\theta$ is the parameterisation given by $p_1 = \theta$, $p_2 = 1-\theta$.

What I would like is to understand the sense in which this is invariant with respect to a coordinate transformation $\theta \to \varphi(\theta)$. To me the term "invariant" would seem to imply something along the lines of $$ \int_{\theta_1}^{\theta_2} \rho(\theta) d \theta = \int_{\varphi(\theta_1)}^{\varphi(\theta_2)} \rho(\varphi(\theta)) d \varphi \qquad\qquad(ii) $$ for any (smooth, differentiable) function $\varphi$ -- but it's easy enough to see that this is not satisfied by the distribution $(i)$ above (and indeed, I doubt there can be any density function that does satisfy this kind of invariance for any transformation). So there must be some other sense intended by "invariant" in this context. I would like to understand this sense in the form of a functional equation similar to $(ii)$, so that I can see how it's satisfied by $(i)$.

Progress

As did points out, the Wikipedia article gives a hint about this, by starting with $$ p(\theta)\propto\sqrt{I(\theta)} $$ and deriving $$ p(\varphi)\propto\sqrt{I(\varphi)} $$ for any smooth function $\varphi(\theta)$. (Note that these equations omit taking the Jacobian of $I$ because they refer to a single-variable case.) Clearly something is invariant here, and it seems like it shouldn't be too hard to express this invariance as a functional equation. However, the more I try to do this the more confused I get. Partly this is because there's just a lot left out of the Wikipedia sketch (e.g. are the constants of proportionality the same in the two equations above, or different? Where is the proof of uniqueness?) but mostly it's because it's really unclear exactly what's being sought, which is why I wanted to express it as a functional equation in the first place.

To reiterate my question, I understand the above equations from Wikipedia, and I can see that they demonstrate an invariance property of some kind. However, I can't see how to express this invariance property in the form of a functional equation similar to $(ii)$, which is what I'm looking for as an answer to this question. I want to first understand the desired invariance property, and then see that the Jeffrey's prior (hopefully uniquely) satisfies it, but the above equations mix up those two steps in a way that I can't see how to separate.

share|improve this question
    
Which part of the question is not dealt with here? –  Did Oct 12 '12 at 22:03
    
This should be posted as a comment rather than an answer, since it is not an answer. However, the link is helpful. To answer your question, the missing bit is the bit where I said "I'd like to understand this sense [of invariance] form of a functional equation similar to (ii), so that I can see how it's satisfied by (i)." Perhaps I can answer this myself now, but if you'd like to post a proper answer detailing it then I'd be happy to award you the bounty. –  Nathaniel Oct 12 '12 at 22:27
    
Sorry but I absolutely completely do not care the least about bounties and points. My answer is written as it is because yes, I believe you can answer this (your)self now. –  Did Oct 12 '12 at 22:33
2  
Perhaps I can, but it seems not at all trivial to me. –  Nathaniel Oct 12 '12 at 23:01
2  
The comments on this question make no sense if you don't already know that @did's comment was originally an answer, which was deleted by a moderator and made into a comment, and that the following two comments were originally comments on did's answer. –  Chris Taylor Oct 18 '12 at 9:14

2 Answers 2

Maybe the problem is that you are forgetting the jacobian of the transformation in (ii).

I suggest that you check carefully the formulas here (hint: $\left| \frac{d \Phi^{- 1}}{d y} \right|$ is the jacobian where $\Phi^{- 1}$ is the inverse transformation). Then, start with some simple examples of some monotonic transformations in order to see the invariance. I suggest to start with $\varphi(\theta)=2\theta$ and $\varphi(\theta)=1-\theta$.

Also, to answer your question, the constants of integration do not matter here. In (i), it is $\pi$. Do the calculations with $\pi$ in there to see that point. Let me know if you are stuck somewhere.

Edit: The dependence on the likelihood is essential for the invariance to hold, because the information is a property of the likelihood and because the object of interest is ultimately the posterior. However, regardless what likelihood you use, the invariance will hold through. This happens through the relationship $ \sqrt{I (\theta)} = \sqrt{I (\varphi (\theta))} | \varphi' (\theta) | $. Indeed this equation links the information of the likelihood to the information of the likelihood given the transformed model. Here $| \varphi' (\theta) |$ is the inverse of the jacobian of the transformation. (I will let you verify this by deriving the information from the likelihood. Just use the chain rule after applying the definition of the information as the expected value of the square of the score). Now, for the prior. \begin{eqnarray*} p (\varphi (\theta) ) & = & \frac{1}{| \varphi' (\theta) |} p (\theta )\\ & = & \frac{1}{| \varphi' (\theta) |} \sqrt{I (\theta)} \\ & = & \sqrt{I (\varphi (\theta))} \\ & = & p (\varphi (\theta)) \end{eqnarray*} The first line is only applying the formula for the jacobian when transforming between posteriors. The second line applies the definition of Jeffreys prior. The third line applies the relationship between the information matrices. The final line applies the definition of Jeffreys prior on $\varphi{(\theta)}$. You can see that the use of Jeffreys prior was essential for $\frac{1}{| \varphi' (\theta) |}$ to cancel out.

Look again at what happens to the posterior ($y$ is obviously the observed sample here) \begin{eqnarray*} p (\varphi (\theta) |y) & = & \frac{1}{| \varphi' (\theta) |} p (\theta |y)\\ & \propto & \frac{1}{| \varphi' (\theta) |} p (\theta) p (y| \theta)\\ & \propto & \frac{1}{| \varphi' (\theta) |} \sqrt{I (\theta)} p (y| \theta)\\ & \propto & \sqrt{I (\varphi (\theta))} |p (y| \theta)\\ & \propto & p (\varphi (\theta)) p (y| \theta) \end{eqnarray*} The only difference is that the second line applies Bayes rule.

As I explained earlier in the comments, it is essential to understand how jacobians work (or differential forms).

share|improve this answer
    
Thanks for the hints. Those equations (quoted from Wikipedia) omit the Jacobian because they refer to the case of a binomial trial, where there is only one variable and the Jacobian of $I$ is just $I$. The problem is not that I don't understand those equations. What I want is to see a definition of the sought invariance property that doesn't refer to the specific form of the Jeffrey's prior, and then see that $\sqrt{|I(\theta)|}$ (hopefully uniquely) satisfies it - and I can't see how to do that. –  Nathaniel Oct 16 '12 at 15:55
    
I still think that your problem is with jacobians and the fact that the formula (ii) is correct for the special case I does not make correct in general. –  Per Oct 17 '12 at 3:41
    
Formula (ii) is not correct in either the special case or in general. What I'm looking for is something like formula (ii), but correct. –  Nathaniel Oct 17 '12 at 11:26
    
zyx's answer is excellent but it uses differential forms. I will add some clarifications to my answer regarding your question about the invariance depending on the likelihood to my answer. –  Per Oct 18 '12 at 7:56
1  
By the way, I don't want to seem obstinate. This is genuinely very helpful, and I'll go through it very carefully later, as well as brushing up on my knowledge of Jacobians in case there's something I've misunderstood. But still, it seems like having a better understanding of how to go from $p(\theta)$ to $p(\varphi(\theta))$ isn't automatically giving me a grasp of what the "XXX" is. I'm fairly certain it's a logical point that I'm missing, rather than something to do with the formal details of the mathematics. –  Nathaniel Oct 18 '12 at 10:53

What is invariant is the volume density $|p_{L_{\theta}}(\theta) dV_{\theta}|$ where $V_\theta$ is the volume form in coordinates $\theta_1, \theta_2, \dots \theta_n$ and $L_\theta$ is the likelihood parametrized by $\theta$. Locally the Fisher matrix $F$ transforms to $(J^{-1})^TFJ^{-1}$ under a change of coordinates with Jacobian $J$, and $\sqrt{\det}$ of this cancels the multiplication of volume forms by $\det J$.

The presentation in Wikipedia is confusing, because

  • the equations are between densities $p(x) dx$, but written as though for the density functions $p()$ that define the priors,

  • the first equality is a claim still to be proven. The following ones are the derivation of that equation.

To read the Wikipedia argument as a chain of equalities of unsigned volume forms, multiply every line by $|d\varphi|$, and use absolute value of all determinants, not the usual signed determinant. Then "$p_{L_{\varphi}}(\varphi) d\varphi \rm{\hskip2pt(claimed)} = p_{L_{\theta}}(\theta) d\theta = (\rm{...Fisher \hskip3pt I \hskip3pt quantities...)} d\varphi = \sqrt{I(\varphi)} d\varphi $.


To answer some of the other questions,

The invariance of $|p dV|$ is the definition of "invariance of prior". Because changes of coordinate alter $dV$, an invariant prior has to depend on more than $p(\theta)$. It is natural to ask for something local on the parameter space, so the invariant prior will be built from a finite number of derivatives of the likelihood evaluated at $\theta$. This means some local finite dimensional linear space of differential quantities at each point with linear maps between the before- and after- coordinate change spaces. Determinants appear because there is a factor of $\det J$ to be killed from the change in $dV$, and because we will want the changes of the local quantities to multiply and cancel each other as is the case in Jeffreys prior, which practically requires a reduction to one dimension where the coordinate change can act on each factor by multiplication by a single number. The Jeffreys prior is a product of two locally defined quantities one of which scales by $\sqrt{A^{-2}}$ and the other by $A$ where $A(\theta)$ is a local factor that depends on $\theta$ and on the coordinate transformation. Computationally it is expressed by Jacobians but only the power-of-$A$ dependences matter and having those cancel out on multiplication. This shows that the invariant prior is very non-unique as there are many other ways to achieve the cancellation. The preference for Jeffreys form of invariant prior is based on other considerations.

share|improve this answer
    
In the univariate case, does the expression in your first sentence reduce to $p(\theta) d\theta$? If so I don't think that can be the thing that's invariant. Since, as you say, $p(\varphi)d\varphi \equiv p(\theta)d\theta$ is an identity, it holds for every pdf $p(\theta)$, not just the Jeffreys prior. –  Nathaniel Oct 17 '12 at 20:51
    
Also, it would help me a lot if you could expand on the distinction you make between "densities $p(x) dx$" and "the density functions $p()$ that define the priors" - I can sort-of see what you mean, but it's not quite clear to me yet. –  Nathaniel Oct 17 '12 at 20:53
    
Finally, whatever the thing that's invariant is, it must surely depend in some way on the likelihood function! –  Nathaniel Oct 17 '12 at 20:55
    
re the second comment, the distinction is between functions and differential forms. What you need for Bayesian statistics (resp., likelihood-based methods) is the ability to integrate against a prior (likelihood), so really $p(x) dx$ is the object of interest. –  zyx Oct 17 '12 at 21:18
    
I made some edits, I think it explains clearly now why the Wikipedia link is not a real answer. –  zyx Oct 18 '12 at 0:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.